Mirrored_Repos/veloren
1
0
Fork 0
mirror of https://gitlab.com/veloren/veloren.git synced 2024-08-30 18:12:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
338dec41ab
veloren/network/Cargo.toml

76 lines
2.2 KiB
TOML
Raw Normal View History

initial implementation of a network api
2019-12-20 13:56:01 +00:00
[package]
COMPLETE REDESIGN of network crate - Implementing a async non-io protocol crate a) no tokio / no channels b) I/O is based on abstraction Sink/Drain c) different Protocols can have a different Drain Type This allow MPSC to send its content without splitting up messages at all! It allows UDP to have internal extra frames to care for security It allows better abstraction for tests Allows benchmarks on the mpsc variant Custom Handshakes to allow sth like Quic protocol easily - reduce the participant managers to 4: channel creations, send, recv and shutdown. keeping the `mut data` in one manager removes the need for all RwLocks. reducing complexity and parallel access problems - more strategic participant shutdown. first send. then wait for remote side to notice recv stop, then remote side will stop send, then local side can stop recv. - metrics are internally abstracted to fit protocol and network layer - in this commit network/protocol tests work and network tests work someway, veloren compiles but does not work - handshake compatible to async_std
2021-01-22 16:09:20 +00:00
name = "veloren-network"
Initial switch to tokio for network, minimum working example.
2021-01-13 13:16:22 +00:00
version = "0.3.0"
initial implementation of a network api
2019-12-20 13:56:01 +00:00
authors = ["Marcel Märtens <marcel.cochem@googlemail.com>"]
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
make `prometheus` optional in network and fix a panic in the server - an extra interface `new_with_regisitry` was created to make sure the interface doesn't depend on the features
2020-07-14 23:34:41 +00:00
[features]
COMPLETE REDESIGN of network crate - Implementing a async non-io protocol crate a) no tokio / no channels b) I/O is based on abstraction Sink/Drain c) different Protocols can have a different Drain Type This allow MPSC to send its content without splitting up messages at all! It allows UDP to have internal extra frames to care for security It allows better abstraction for tests Allows benchmarks on the mpsc variant Custom Handshakes to allow sth like Quic protocol easily - reduce the participant managers to 4: channel creations, send, recv and shutdown. keeping the `mut data` in one manager removes the need for all RwLocks. reducing complexity and parallel access problems - more strategic participant shutdown. first send. then wait for remote side to notice recv stop, then remote side will stop send, then local side can stop recv. - metrics are internally abstracted to fit protocol and network layer - in this commit network/protocol tests work and network tests work someway, veloren compiles but does not work - handshake compatible to async_std
2021-01-22 16:09:20 +00:00
metrics = ["prometheus", "network-protocol/metrics"]
Propper Compression support of network. - Compression is no longer enabled always but can now be enabled per Stream. If a Stream is Compression enabled it will compress and decompress all msg (except for `raw` access) before handling them internally. You need to handle compression yourself for `raw` fn. - added a new feature to the network crate to enable or disable the compression - switched to `lz-fear` instead of `lz4-compression` - use `bitflags` to represent the `Promises` struct
2020-08-25 13:32:42 +00:00
compression = ["lz-fear"]
Quic: We had the followuing problem: - locally we open a stream, our local Drain is sending OpenStream - remote Sink will know this and notify remote Drain - remote side sends a message - local sink does not know about the Stream. as there is (and CANT) be a wat to notify local Sink from local Drain (it could introduce race conditions). One of the possible solutions was, that the remote drain will copy the OpenStream Msg ON the Quic::stream before first data is send. This would work but is complicated. Instead we now just mark such streams as "potentially open" and we listen for the first DataHeader to get it's SID. add support for unreliable messages in quic protocol, benchmarks
2021-04-09 11:17:38 +00:00
quic = ["quinn"]
make `prometheus` optional in network and fix a panic in the server - an extra interface `new_with_regisitry` was created to make sure the interface doesn't depend on the features
2020-07-14 23:34:41 +00:00
work on getting quic in the network
2021-04-11 21:37:48 +00:00
default = ["metrics","compression","quic"]
make `prometheus` optional in network and fix a panic in the server - an extra interface `new_with_regisitry` was created to make sure the interface doesn't depend on the features
2020-07-14 23:34:41 +00:00
initial implementation of a network api
2019-12-20 13:56:01 +00:00
[dependencies]
Great improvements to the codebase: - better logging in network - we now notify the send of what happened in recv in participant. - works with veloren master servers - works in singleplayer, using a actual mid. - add `mpsc` in whole stack incl tests - speed up internal read/write with `Bytes` crate - use `prometheus-hyper` for metrics - use a metrics cache
2021-02-10 10:37:42 +00:00
network-protocol = { package = "veloren-network-protocol", path = "protocol" }
COMPLETE REDESIGN of network crate - Implementing a async non-io protocol crate a) no tokio / no channels b) I/O is based on abstraction Sink/Drain c) different Protocols can have a different Drain Type This allow MPSC to send its content without splitting up messages at all! It allows UDP to have internal extra frames to care for security It allows better abstraction for tests Allows benchmarks on the mpsc variant Custom Handshakes to allow sth like Quic protocol easily - reduce the participant managers to 4: channel creations, send, recv and shutdown. keeping the `mut data` in one manager removes the need for all RwLocks. reducing complexity and parallel access problems - more strategic participant shutdown. first send. then wait for remote side to notice recv stop, then remote side will stop send, then local side can stop recv. - metrics are internally abstracted to fit protocol and network layer - in this commit network/protocol tests work and network tests work someway, veloren compiles but does not work - handshake compatible to async_std
2021-01-22 16:09:20 +00:00
Code/Dependency Cleanup
2020-03-04 15:52:30 +00:00
#serialisation
Fixed torvus
2021-05-28 00:56:16 +00:00
bincode = "1.3.2"
switch from serde to manually for speed, remove async_serde - removing async_serde as it seems to be not usefull the idea was because deserialising is slow parallising it could speed up. Whoever we need to keep the order of frames, (at least for controlframes) so serialising in threads would be quite complicated. Also serialisation is quite fast, about 1 Gbit/s such speed is enough for messaging, it's more important to serve parallel streams better. Thats why i am removing async serde coding for now - frames are no longer serialized by serde, by byte by byte manually, increadible speed upgrade - more metrics - switch channel_creator into for_each_concurrent - removing some pool.spwan_ok() as they dont allow me to use self - reduce features needed
2020-04-24 10:56:04 +00:00
serde = { version = "1.0" }
Code/Dependency Cleanup
2020-03-04 15:52:30 +00:00
#sending
update chrossbeam, tracy, prometheus (and reduce server deps to crossbeam-channel)
2020-12-15 23:51:07 +00:00
crossbeam-channel = "0.5"
update other binaries
2021-11-19 14:40:59 +00:00
tokio = { version = "1.14", default-features = false, features = ["io-util", "macros", "rt", "net", "time"] }
get rid of `async_std::channel` switch to `tokio` and `async_channel` crate. I wanted to do tokio first, but it doesnt feature Sender::close(), thus i included async_channel Got rid of `futures` and only need `futures_core` and `futures_util`. Tokio does not support `Stream` and `StreamExt` so for now i need to use `tokio-stream`, i think this will go in `std` in the future Created `b2b_close_stream_opened_sender_r` as the shutdown procedure does not need a copy of a Sender, it just need to stop it. Various adjustments, e.g. for `select!` which now requieres a `&mut` for oneshots. Future things to do: - Use some better signalling than oneshot<()> in some cases. - Use a Watch for the Prio propergation (impl. it ofc) - Use Bounded Channels in order to improve performance - adjust tests coding bring tests to work
2021-01-15 13:04:32 +00:00
tokio-stream = { version = "0.1.2", default-features = false }
Code/Dependency Cleanup
2020-03-04 15:52:30 +00:00
#tracing and metrics
Great improvements to the codebase: - better logging in network - we now notify the send of what happened in recv in participant. - works with veloren master servers - works in singleplayer, using a actual mid. - add `mpsc` in whole stack incl tests - speed up internal read/write with `Bytes` crate - use `prometheus-hyper` for metrics - use a metrics cache
2021-02-10 10:37:42 +00:00
tracing = { version = "0.1", default-features = false, features = ["attributes"]}
update dependencies, including removal of some tracy deps as they are get through common/tracy
2021-03-09 19:12:57 +00:00
prometheus = { version = "0.12", default-features = false, optional = true }
Code/Dependency Cleanup
2020-03-04 15:52:30 +00:00
#async
get rid of `async_std::channel` switch to `tokio` and `async_channel` crate. I wanted to do tokio first, but it doesnt feature Sender::close(), thus i included async_channel Got rid of `futures` and only need `futures_core` and `futures_util`. Tokio does not support `Stream` and `StreamExt` so for now i need to use `tokio-stream`, i think this will go in `std` in the future Created `b2b_close_stream_opened_sender_r` as the shutdown procedure does not need a copy of a Sender, it just need to stop it. Various adjustments, e.g. for `select!` which now requieres a `&mut` for oneshots. Future things to do: - Use some better signalling than oneshot<()> in some cases. - Use a Watch for the Prio propergation (impl. it ofc) - Use Bounded Channels in order to improve performance - adjust tests coding bring tests to work
2021-01-15 13:04:32 +00:00
futures-core = { version = "0.3", default-features = false }
Update Network Protocol - now last digit version is compatible 0.6.0 will connect to 0.6.1 - the TCP DATA Frames no longer contain START field, as it's not needed - the TCP OPENSTREAM Frames will now contain the BANDWIDTH field - MID is not Protocol internal Update network - update API with Bandwidth Update veloren - introduce better runtime and `async` things that are IO bound. - Remove `uvth` and instead use `tokio::runtime::Runtime::spawn_blocking` - remove futures_execute from client and server use tokio::runtime::Runtime instead - give threads a Name
2021-02-18 00:01:57 +00:00
futures-util = { version = "0.3.7", default-features = false, features = ["std"] }
get rid of `async_std::channel` switch to `tokio` and `async_channel` crate. I wanted to do tokio first, but it doesnt feature Sender::close(), thus i included async_channel Got rid of `futures` and only need `futures_core` and `futures_util`. Tokio does not support `Stream` and `StreamExt` so for now i need to use `tokio-stream`, i think this will go in `std` in the future Created `b2b_close_stream_opened_sender_r` as the shutdown procedure does not need a copy of a Sender, it just need to stop it. Various adjustments, e.g. for `select!` which now requieres a `&mut` for oneshots. Future things to do: - Use some better signalling than oneshot<()> in some cases. - Use a Watch for the Prio propergation (impl. it ofc) - Use Bounded Channels in order to improve performance - adjust tests coding bring tests to work
2021-01-15 13:04:32 +00:00
async-channel = "1.5.1" #use for .close() channels
shutdown and udp/mpsc - theorectically closing of streams and shutdown - mpsc and udp preparations - cleanup and build better tests
2020-03-10 00:07:36 +00:00
#mpsc channel registry
switch from serde to manually for speed, remove async_serde - removing async_serde as it seems to be not usefull the idea was because deserialising is slow parallising it could speed up. Whoever we need to keep the order of frames, (at least for controlframes) so serialising in threads would be quite complicated. Also serialisation is quite fast, about 1 Gbit/s such speed is enough for messaging, it's more important to serve parallel streams better. Thats why i am removing async serde coding for now - frames are no longer serialized by serde, by byte by byte manually, increadible speed upgrade - more metrics - switch channel_creator into for_each_concurrent - removing some pool.spwan_ok() as they dont allow me to use self - reduce features needed
2020-04-24 10:56:04 +00:00
lazy_static = { version = "1.4", default-features = false }
Update rand dependency
2021-01-26 20:23:18 +00:00
rand = { version = "0.8" }
Quic: We had the followuing problem: - locally we open a stream, our local Drain is sending OpenStream - remote Sink will know this and notify remote Drain - remote side sends a message - local sink does not know about the Stream. as there is (and CANT) be a wat to notify local Sink from local Drain (it could introduce race conditions). One of the possible solutions was, that the remote drain will copy the OpenStream Msg ON the Quic::stream before first data is send. This would work but is complicated. Instead we now just mark such streams as "potentially open" and we listen for the first DataHeader to get it's SID. add support for unreliable messages in quic protocol, benchmarks
2021-04-09 11:17:38 +00:00
#quic support
updating dependencies, cannot update the following dependencies: - vek: Sharps SIMD isnt upstream - tracing-subscriber: MakeWriter was adjusted and i was to lazy to fiddle with lifetimes, - refinery, rustsql: we have a custom refinery version which is incompatible with newer rustsql - equi + egui_winit + egui_wgpu_backend: i tried it in this commit but it turned out that they dependo n wgpu which we cant update - wgpu: cant update due new version doesnt support DX11 Got quinn updated which now require some dependencies to be explicit.
2021-11-19 13:01:08 +00:00
quinn = { version = "0.8", optional = true }
update other binaries
2021-11-19 14:40:59 +00:00
rustls = "0.20.1"
Propper Compression support of network. - Compression is no longer enabled always but can now be enabled per Stream. If a Stream is Compression enabled it will compress and decompress all msg (except for `raw` access) before handling them internally. You need to handle compression yourself for `raw` fn. - added a new feature to the network crate to enable or disable the compression - switched to `lz-fear` instead of `lz4-compression` - use `bitflags` to represent the `Promises` struct
2020-08-25 13:32:42 +00:00
#stream flags
bitflags = "1.2.1"
lz-fear = { version = "0.1.1", optional = true }
COMPLETE REDESIGN of network crate - Implementing a async non-io protocol crate a) no tokio / no channels b) I/O is based on abstraction Sink/Drain c) different Protocols can have a different Drain Type This allow MPSC to send its content without splitting up messages at all! It allows UDP to have internal extra frames to care for security It allows better abstraction for tests Allows benchmarks on the mpsc variant Custom Handshakes to allow sth like Quic protocol easily - reduce the participant managers to 4: channel creations, send, recv and shutdown. keeping the `mut data` in one manager removes the need for all RwLocks. reducing complexity and parallel access problems - more strategic participant shutdown. first send. then wait for remote side to notice recv stop, then remote side will stop send, then local side can stop recv. - metrics are internally abstracted to fit protocol and network layer - in this commit network/protocol tests work and network tests work someway, veloren compiles but does not work - handshake compatible to async_std
2021-01-22 16:09:20 +00:00
# async traits
async-trait = "0.1.42"
Great improvements to the codebase: - better logging in network - we now notify the send of what happened in recv in participant. - works with veloren master servers - works in singleplayer, using a actual mid. - add `mpsc` in whole stack incl tests - speed up internal read/write with `Bytes` crate - use `prometheus-hyper` for metrics - use a metrics cache
2021-02-10 10:37:42 +00:00
bytes = "^1"
fix error handling in networking and switch to hashbrown, fixing #1118
2021-05-04 13:27:30 +00:00
# faster HashMaps
hashbrown = { version = ">=0.9, <0.12" }
Add a speedtest program to benchmark networking
2020-02-25 18:30:50 +00:00
[dev-dependencies]
Bump tracing-subscriber to 0.3.2 We were pulling two versions of it. Causing some compilation errors
2021-11-26 17:41:51 +00:00
tracing-subscriber = { version = "0.3.2", default-features = false, features = ["env-filter", "fmt", "time", "ansi", "smallvec"] }
update other binaries
2021-11-19 14:40:59 +00:00
tokio = { version = "1.14", default-features = false, features = ["io-std", "fs", "rt-multi-thread"] }
Update Network Protocol - now last digit version is compatible 0.6.0 will connect to 0.6.1 - the TCP DATA Frames no longer contain START field, as it's not needed - the TCP OPENSTREAM Frames will now contain the BANDWIDTH field - MID is not Protocol internal Update network - update API with Bandwidth Update veloren - introduce better runtime and `async` things that are IO bound. - Remove `uvth` and instead use `tokio::runtime::Runtime::spawn_blocking` - remove futures_execute from client and server use tokio::runtime::Runtime instead - give threads a Name
2021-02-18 00:01:57 +00:00
futures-util = { version = "0.3.7", default-features = false, features = ["sink", "std"] }
Switch veloren_network over to use the official example layout. adjusted those examples to run again created a CI TEST to always `check` examples fixed fmt in examples so that pipeline gets green
2020-08-26 08:07:22 +00:00
clap = { version = "2.33", default-features = false }
shellexpand = "2.0.0"
update a few deps
2021-01-20 14:53:58 +00:00
serde = { version = "1.0", features = ["derive"] }
update dependencies, including removal of some tracy deps as they are get through common/tracy
2021-03-09 19:12:57 +00:00
prometheus-hyper = "0.1.2"
Stabelize Network again: - completly switch to Bytes, even in api. speed up TCP by fak 2 - improve benchmarks - speed up mpsc metrics - gracefully handle shutdown by interpreting Ok(0) as tokio::tcpstream closed now. - fix hotloop in participants by adding `Some(n)` to fix endless handing. - fix closing bug by closing streams after `recv_mgr` is shutdown even if now shutdown is triggered locally. - fix prometheus - no longer throw when a `Stream` is dropped while participant still receives a msg for it. - fix the bandwith handling, TCP network send speed is up to 1.5GiB/s while recv is 150MiB/s - add documentation - tmp require rt-multi-threaded in client for tokio, to not fail cargo check this is prob stable, i tested over 1 hour. after that some optimisations in priomgr. and impl. propper bandwith. Speed is up to 2GB/s write and 150MB/s recv on a single core sync add documentation
2021-02-14 17:45:12 +00:00
criterion = { version = "0.3.4", features = ["default", "async_tokio"] }
protocoladdr change for listen and connect (remove a loop in quic protocol which wasnt a actual loop)
2021-04-15 08:16:42 +00:00
#quic
rcgen = { version = "0.8.10"}
Stabelize Network again: - completly switch to Bytes, even in api. speed up TCP by fak 2 - improve benchmarks - speed up mpsc metrics - gracefully handle shutdown by interpreting Ok(0) as tokio::tcpstream closed now. - fix hotloop in participants by adding `Some(n)` to fix endless handing. - fix closing bug by closing streams after `recv_mgr` is shutdown even if now shutdown is triggered locally. - fix prometheus - no longer throw when a `Stream` is dropped while participant still receives a msg for it. - fix the bandwith handling, TCP network send speed is up to 1.5GiB/s while recv is 150MiB/s - add documentation - tmp require rt-multi-threaded in client for tokio, to not fail cargo check this is prob stable, i tested over 1 hour. after that some optimisations in priomgr. and impl. propper bandwith. Speed is up to 2GB/s write and 150MB/s recv on a single core sync add documentation
2021-02-14 17:45:12 +00:00
[[bench]]
name = "speed"
harness = false
Great improvements to the codebase: - better logging in network - we now notify the send of what happened in recv in participant. - works with veloren master servers - works in singleplayer, using a actual mid. - add `mpsc` in whole stack incl tests - speed up internal read/write with `Bytes` crate - use `prometheus-hyper` for metrics - use a metrics cache
2021-02-10 10:37:42 +00:00
[[example]]
name = "fileshare"
[[example]]
name = "network-speed"
[[example]]
name = "chat"
[[example]]
name = "tcp_loadtest"
Reference in New Issue Copy Permalink