Mirrored_Repos/veloren
1
0
Fork 0
mirror of https://gitlab.com/veloren/veloren.git synced 2024-08-30 18:12:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
27fb8a9f6c
veloren/network/Cargo.toml

68 lines
2.0 KiB
TOML
Raw Normal View History

initial implementation of a network api
2019-12-20 13:56:01 +00:00
[package]
COMPLETE REDESIGN of network crate - Implementing a async non-io protocol crate a) no tokio / no channels b) I/O is based on abstraction Sink/Drain c) different Protocols can have a different Drain Type This allow MPSC to send its content without splitting up messages at all! It allows UDP to have internal extra frames to care for security It allows better abstraction for tests Allows benchmarks on the mpsc variant Custom Handshakes to allow sth like Quic protocol easily - reduce the participant managers to 4: channel creations, send, recv and shutdown. keeping the `mut data` in one manager removes the need for all RwLocks. reducing complexity and parallel access problems - more strategic participant shutdown. first send. then wait for remote side to notice recv stop, then remote side will stop send, then local side can stop recv. - metrics are internally abstracted to fit protocol and network layer - in this commit network/protocol tests work and network tests work someway, veloren compiles but does not work - handshake compatible to async_std
2021-01-22 16:09:20 +00:00
name = "veloren-network"
Initial switch to tokio for network, minimum working example.
2021-01-13 13:16:22 +00:00
version = "0.3.0"
initial implementation of a network api
2019-12-20 13:56:01 +00:00
authors = ["Marcel Märtens <marcel.cochem@googlemail.com>"]
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
make `prometheus` optional in network and fix a panic in the server - an extra interface `new_with_regisitry` was created to make sure the interface doesn't depend on the features
2020-07-14 23:34:41 +00:00
[features]
COMPLETE REDESIGN of network crate - Implementing a async non-io protocol crate a) no tokio / no channels b) I/O is based on abstraction Sink/Drain c) different Protocols can have a different Drain Type This allow MPSC to send its content without splitting up messages at all! It allows UDP to have internal extra frames to care for security It allows better abstraction for tests Allows benchmarks on the mpsc variant Custom Handshakes to allow sth like Quic protocol easily - reduce the participant managers to 4: channel creations, send, recv and shutdown. keeping the `mut data` in one manager removes the need for all RwLocks. reducing complexity and parallel access problems - more strategic participant shutdown. first send. then wait for remote side to notice recv stop, then remote side will stop send, then local side can stop recv. - metrics are internally abstracted to fit protocol and network layer - in this commit network/protocol tests work and network tests work someway, veloren compiles but does not work - handshake compatible to async_std
2021-01-22 16:09:20 +00:00
metrics = ["prometheus", "network-protocol/metrics"]
Propper Compression support of network. - Compression is no longer enabled always but can now be enabled per Stream. If a Stream is Compression enabled it will compress and decompress all msg (except for `raw` access) before handling them internally. You need to handle compression yourself for `raw` fn. - added a new feature to the network crate to enable or disable the compression - switched to `lz-fear` instead of `lz4-compression` - use `bitflags` to represent the `Promises` struct
2020-08-25 13:32:42 +00:00
compression = ["lz-fear"]
make `prometheus` optional in network and fix a panic in the server - an extra interface `new_with_regisitry` was created to make sure the interface doesn't depend on the features
2020-07-14 23:34:41 +00:00
Propper Compression support of network. - Compression is no longer enabled always but can now be enabled per Stream. If a Stream is Compression enabled it will compress and decompress all msg (except for `raw` access) before handling them internally. You need to handle compression yourself for `raw` fn. - added a new feature to the network crate to enable or disable the compression - switched to `lz-fear` instead of `lz4-compression` - use `bitflags` to represent the `Promises` struct
2020-08-25 13:32:42 +00:00
default = ["metrics","compression"]
make `prometheus` optional in network and fix a panic in the server - an extra interface `new_with_regisitry` was created to make sure the interface doesn't depend on the features
2020-07-14 23:34:41 +00:00
initial implementation of a network api
2019-12-20 13:56:01 +00:00
[dependencies]
Great improvements to the codebase: - better logging in network - we now notify the send of what happened in recv in participant. - works with veloren master servers - works in singleplayer, using a actual mid. - add `mpsc` in whole stack incl tests - speed up internal read/write with `Bytes` crate - use `prometheus-hyper` for metrics - use a metrics cache
2021-02-10 10:37:42 +00:00
network-protocol = { package = "veloren-network-protocol", path = "protocol" }
COMPLETE REDESIGN of network crate - Implementing a async non-io protocol crate a) no tokio / no channels b) I/O is based on abstraction Sink/Drain c) different Protocols can have a different Drain Type This allow MPSC to send its content without splitting up messages at all! It allows UDP to have internal extra frames to care for security It allows better abstraction for tests Allows benchmarks on the mpsc variant Custom Handshakes to allow sth like Quic protocol easily - reduce the participant managers to 4: channel creations, send, recv and shutdown. keeping the `mut data` in one manager removes the need for all RwLocks. reducing complexity and parallel access problems - more strategic participant shutdown. first send. then wait for remote side to notice recv stop, then remote side will stop send, then local side can stop recv. - metrics are internally abstracted to fit protocol and network layer - in this commit network/protocol tests work and network tests work someway, veloren compiles but does not work - handshake compatible to async_std
2021-01-22 16:09:20 +00:00
Code/Dependency Cleanup
2020-03-04 15:52:30 +00:00
#serialisation
update a few deps
2021-01-20 14:53:58 +00:00
bincode = "1.3.1"
switch from serde to manually for speed, remove async_serde - removing async_serde as it seems to be not usefull the idea was because deserialising is slow parallising it could speed up. Whoever we need to keep the order of frames, (at least for controlframes) so serialising in threads would be quite complicated. Also serialisation is quite fast, about 1 Gbit/s such speed is enough for messaging, it's more important to serve parallel streams better. Thats why i am removing async serde coding for now - frames are no longer serialized by serde, by byte by byte manually, increadible speed upgrade - more metrics - switch channel_creator into for_each_concurrent - removing some pool.spwan_ok() as they dont allow me to use self - reduce features needed
2020-04-24 10:56:04 +00:00
serde = { version = "1.0" }
Code/Dependency Cleanup
2020-03-04 15:52:30 +00:00
#sending
update chrossbeam, tracy, prometheus (and reduce server deps to crossbeam-channel)
2020-12-15 23:51:07 +00:00
crossbeam-channel = "0.5"
Initial switch to tokio for network, minimum working example.
2021-01-13 13:16:22 +00:00
tokio = { version = "1.2", default-features = false, features = ["io-util", "macros", "rt", "net", "time"] }
get rid of `async_std::channel` switch to `tokio` and `async_channel` crate. I wanted to do tokio first, but it doesnt feature Sender::close(), thus i included async_channel Got rid of `futures` and only need `futures_core` and `futures_util`. Tokio does not support `Stream` and `StreamExt` so for now i need to use `tokio-stream`, i think this will go in `std` in the future Created `b2b_close_stream_opened_sender_r` as the shutdown procedure does not need a copy of a Sender, it just need to stop it. Various adjustments, e.g. for `select!` which now requieres a `&mut` for oneshots. Future things to do: - Use some better signalling than oneshot<()> in some cases. - Use a Watch for the Prio propergation (impl. it ofc) - Use Bounded Channels in order to improve performance - adjust tests coding bring tests to work
2021-01-15 13:04:32 +00:00
tokio-stream = { version = "0.1.2", default-features = false }
Code/Dependency Cleanup
2020-03-04 15:52:30 +00:00
#tracing and metrics
Great improvements to the codebase: - better logging in network - we now notify the send of what happened in recv in participant. - works with veloren master servers - works in singleplayer, using a actual mid. - add `mpsc` in whole stack incl tests - speed up internal read/write with `Bytes` crate - use `prometheus-hyper` for metrics - use a metrics cache
2021-02-10 10:37:42 +00:00
tracing = { version = "0.1", default-features = false, features = ["attributes"]}
update dependencies, including removal of some tracy deps as they are get through common/tracy
2021-03-09 19:12:57 +00:00
prometheus = { version = "0.12", default-features = false, optional = true }
Code/Dependency Cleanup
2020-03-04 15:52:30 +00:00
#async
get rid of `async_std::channel` switch to `tokio` and `async_channel` crate. I wanted to do tokio first, but it doesnt feature Sender::close(), thus i included async_channel Got rid of `futures` and only need `futures_core` and `futures_util`. Tokio does not support `Stream` and `StreamExt` so for now i need to use `tokio-stream`, i think this will go in `std` in the future Created `b2b_close_stream_opened_sender_r` as the shutdown procedure does not need a copy of a Sender, it just need to stop it. Various adjustments, e.g. for `select!` which now requieres a `&mut` for oneshots. Future things to do: - Use some better signalling than oneshot<()> in some cases. - Use a Watch for the Prio propergation (impl. it ofc) - Use Bounded Channels in order to improve performance - adjust tests coding bring tests to work
2021-01-15 13:04:32 +00:00
futures-core = { version = "0.3", default-features = false }
Update Network Protocol - now last digit version is compatible 0.6.0 will connect to 0.6.1 - the TCP DATA Frames no longer contain START field, as it's not needed - the TCP OPENSTREAM Frames will now contain the BANDWIDTH field - MID is not Protocol internal Update network - update API with Bandwidth Update veloren - introduce better runtime and `async` things that are IO bound. - Remove `uvth` and instead use `tokio::runtime::Runtime::spawn_blocking` - remove futures_execute from client and server use tokio::runtime::Runtime instead - give threads a Name
2021-02-18 00:01:57 +00:00
futures-util = { version = "0.3.7", default-features = false, features = ["std"] }
get rid of `async_std::channel` switch to `tokio` and `async_channel` crate. I wanted to do tokio first, but it doesnt feature Sender::close(), thus i included async_channel Got rid of `futures` and only need `futures_core` and `futures_util`. Tokio does not support `Stream` and `StreamExt` so for now i need to use `tokio-stream`, i think this will go in `std` in the future Created `b2b_close_stream_opened_sender_r` as the shutdown procedure does not need a copy of a Sender, it just need to stop it. Various adjustments, e.g. for `select!` which now requieres a `&mut` for oneshots. Future things to do: - Use some better signalling than oneshot<()> in some cases. - Use a Watch for the Prio propergation (impl. it ofc) - Use Bounded Channels in order to improve performance - adjust tests coding bring tests to work
2021-01-15 13:04:32 +00:00
async-channel = "1.5.1" #use for .close() channels
shutdown and udp/mpsc - theorectically closing of streams and shutdown - mpsc and udp preparations - cleanup and build better tests
2020-03-10 00:07:36 +00:00
#mpsc channel registry
switch from serde to manually for speed, remove async_serde - removing async_serde as it seems to be not usefull the idea was because deserialising is slow parallising it could speed up. Whoever we need to keep the order of frames, (at least for controlframes) so serialising in threads would be quite complicated. Also serialisation is quite fast, about 1 Gbit/s such speed is enough for messaging, it's more important to serve parallel streams better. Thats why i am removing async serde coding for now - frames are no longer serialized by serde, by byte by byte manually, increadible speed upgrade - more metrics - switch channel_creator into for_each_concurrent - removing some pool.spwan_ok() as they dont allow me to use self - reduce features needed
2020-04-24 10:56:04 +00:00
lazy_static = { version = "1.4", default-features = false }
Update rand dependency
2021-01-26 20:23:18 +00:00
rand = { version = "0.8" }
Propper Compression support of network. - Compression is no longer enabled always but can now be enabled per Stream. If a Stream is Compression enabled it will compress and decompress all msg (except for `raw` access) before handling them internally. You need to handle compression yourself for `raw` fn. - added a new feature to the network crate to enable or disable the compression - switched to `lz-fear` instead of `lz4-compression` - use `bitflags` to represent the `Promises` struct
2020-08-25 13:32:42 +00:00
#stream flags
bitflags = "1.2.1"
lz-fear = { version = "0.1.1", optional = true }
COMPLETE REDESIGN of network crate - Implementing a async non-io protocol crate a) no tokio / no channels b) I/O is based on abstraction Sink/Drain c) different Protocols can have a different Drain Type This allow MPSC to send its content without splitting up messages at all! It allows UDP to have internal extra frames to care for security It allows better abstraction for tests Allows benchmarks on the mpsc variant Custom Handshakes to allow sth like Quic protocol easily - reduce the participant managers to 4: channel creations, send, recv and shutdown. keeping the `mut data` in one manager removes the need for all RwLocks. reducing complexity and parallel access problems - more strategic participant shutdown. first send. then wait for remote side to notice recv stop, then remote side will stop send, then local side can stop recv. - metrics are internally abstracted to fit protocol and network layer - in this commit network/protocol tests work and network tests work someway, veloren compiles but does not work - handshake compatible to async_std
2021-01-22 16:09:20 +00:00
# async traits
async-trait = "0.1.42"
Great improvements to the codebase: - better logging in network - we now notify the send of what happened in recv in participant. - works with veloren master servers - works in singleplayer, using a actual mid. - add `mpsc` in whole stack incl tests - speed up internal read/write with `Bytes` crate - use `prometheus-hyper` for metrics - use a metrics cache
2021-02-10 10:37:42 +00:00
bytes = "^1"
Add a speedtest program to benchmark networking
2020-02-25 18:30:50 +00:00
[dev-dependencies]
StreamError::Deserialize is now triggered when `recv` fails because of wrong type - added PartialEq to StreamError for test purposes (only yet!) - removed async_recv example as it's no longer for any use. It was created before the COMPLETE REWRITE in order to verify that my own async interface on top of mio works. However it's now guaranteed by async-std and futures. no need for a special test - remove uvth from dependencies and replace it with a `FnOnce` - fix ALL clippy (network) lints - basic fix for a channel drop scenario: TODO: this needs some further fixes up to know only destruction of participant by api was covered correctly. we had an issue when the underlying channels got dropped. So now we have a participant without channels. We need to buffer the requests and try to reopen a channel ASAP! If no channel could be reopened we need to close the Participant, while a) leaving the BParticipant in takt, knowing that it only waits for a propper close by scheduler b) close the BParticipant gracefully. Notifying the scheduler to remove its stuff (either scheduler schould detect a stopped BParticipant or BParticipant will send Scheduler it's own destruction, and then Scheduler just does the same like when API forces a close) Keep the Participant alive and wait for the api to acces BParticipant to notice it's closed and then wait for a disconnect which isn't doing anything as it was already cleaned up in the background
2020-06-08 09:47:39 +00:00
tracing-subscriber = { version = "0.2.3", default-features = false, features = ["env-filter", "fmt", "chrono", "ansi", "smallvec"] }
Great improvements to the codebase: - better logging in network - we now notify the send of what happened in recv in participant. - works with veloren master servers - works in singleplayer, using a actual mid. - add `mpsc` in whole stack incl tests - speed up internal read/write with `Bytes` crate - use `prometheus-hyper` for metrics - use a metrics cache
2021-02-10 10:37:42 +00:00
tokio = { version = "1.2", default-features = false, features = ["io-std", "fs", "rt-multi-thread"] }
Update Network Protocol - now last digit version is compatible 0.6.0 will connect to 0.6.1 - the TCP DATA Frames no longer contain START field, as it's not needed - the TCP OPENSTREAM Frames will now contain the BANDWIDTH field - MID is not Protocol internal Update network - update API with Bandwidth Update veloren - introduce better runtime and `async` things that are IO bound. - Remove `uvth` and instead use `tokio::runtime::Runtime::spawn_blocking` - remove futures_execute from client and server use tokio::runtime::Runtime instead - give threads a Name
2021-02-18 00:01:57 +00:00
futures-util = { version = "0.3.7", default-features = false, features = ["sink", "std"] }
Switch veloren_network over to use the official example layout. adjusted those examples to run again created a CI TEST to always `check` examples fixed fmt in examples so that pipeline gets green
2020-08-26 08:07:22 +00:00
clap = { version = "2.33", default-features = false }
shellexpand = "2.0.0"
update a few deps
2021-01-20 14:53:58 +00:00
serde = { version = "1.0", features = ["derive"] }
update dependencies, including removal of some tracy deps as they are get through common/tracy
2021-03-09 19:12:57 +00:00
prometheus-hyper = "0.1.2"
Stabelize Network again: - completly switch to Bytes, even in api. speed up TCP by fak 2 - improve benchmarks - speed up mpsc metrics - gracefully handle shutdown by interpreting Ok(0) as tokio::tcpstream closed now. - fix hotloop in participants by adding `Some(n)` to fix endless handing. - fix closing bug by closing streams after `recv_mgr` is shutdown even if now shutdown is triggered locally. - fix prometheus - no longer throw when a `Stream` is dropped while participant still receives a msg for it. - fix the bandwith handling, TCP network send speed is up to 1.5GiB/s while recv is 150MiB/s - add documentation - tmp require rt-multi-threaded in client for tokio, to not fail cargo check this is prob stable, i tested over 1 hour. after that some optimisations in priomgr. and impl. propper bandwith. Speed is up to 2GB/s write and 150MB/s recv on a single core sync add documentation
2021-02-14 17:45:12 +00:00
criterion = { version = "0.3.4", features = ["default", "async_tokio"] }
[[bench]]
name = "speed"
harness = false
Great improvements to the codebase: - better logging in network - we now notify the send of what happened in recv in participant. - works with veloren master servers - works in singleplayer, using a actual mid. - add `mpsc` in whole stack incl tests - speed up internal read/write with `Bytes` crate - use `prometheus-hyper` for metrics - use a metrics cache
2021-02-10 10:37:42 +00:00
[[example]]
name = "fileshare"
[[example]]
name = "network-speed"
[[example]]
name = "chat"
[[example]]
name = "tcp_loadtest"
Reference in New Issue Copy Permalink