mirror of
https://gitlab.com/veloren/veloren.git
synced 2024-08-30 18:12:32 +00:00
9884019963
25 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Marcel Märtens
|
9884019963 |
COMPLETE REDESIGN of network crate
- Implementing a async non-io protocol crate
a) no tokio / no channels
b) I/O is based on abstraction Sink/Drain
c) different Protocols can have a different Drain Type
This allow MPSC to send its content without splitting up messages at all!
It allows UDP to have internal extra frames to care for security
It allows better abstraction for tests
Allows benchmarks on the mpsc variant
Custom Handshakes to allow sth like Quic protocol easily
- reduce the participant managers to 4: channel creations, send, recv and shutdown.
keeping the `mut data` in one manager removes the need for all RwLocks.
reducing complexity and parallel access problems
- more strategic participant shutdown. first send. then wait for remote side to notice recv stop, then remote side will stop send, then local side can stop recv.
- metrics are internally abstracted to fit protocol and network layer
- in this commit network/protocol tests work and network tests work someway, veloren compiles but does not work
- handshake compatible to async_std
|
||
|
Marcel Märtens
|
5aa1940ef8 |
get rid of async_std::channel
switch to `tokio` and `async_channel` crate. I wanted to do tokio first, but it doesnt feature Sender::close(), thus i included async_channel Got rid of `futures` and only need `futures_core` and `futures_util`. Tokio does not support `Stream` and `StreamExt` so for now i need to use `tokio-stream`, i think this will go in `std` in the future Created `b2b_close_stream_opened_sender_r` as the shutdown procedure does not need a copy of a Sender, it just need to stop it. Various adjustments, e.g. for `select!` which now requieres a `&mut` for oneshots. Future things to do: - Use some better signalling than oneshot<()> in some cases. - Use a Watch for the Prio propergation (impl. it ofc) - Use Bounded Channels in order to improve performance - adjust tests coding bring tests to work |
||
jiminycrick
|
8b97199245 | Update rand dependency | ||
|
Marcel Märtens
|
b5f48014a9 |
Streams no longer panic when recv on a StreamClosed Stream. Panicing is a "feature" of futures::channel
Refactor the `send_raw` and `recv_raw` completly. We now expost `Message` which has a public `serialize` and `deseialize` fn for the first time. This makes using the `raw` methods of a stream much easier and is a requierement for using "copy_less" sending to multiple streams |
||
|
Marcel Märtens
|
144f88f811 |
Propper Compression support of network.
- Compression is no longer enabled always but can now be enabled per Stream. If a Stream is Compression enabled it will compress and decompress all msg (except for `raw` access) before handling them internally. You need to handle compression yourself for `raw` fn. - added a new feature to the network crate to enable or disable the compression - switched to `lz-fear` instead of `lz4-compression` - use `bitflags` to represent the `Promises` struct |
||
notoria
|
2be4202d01 | Corrected some spelling errors | ||
Joshua Yanovski
|
2dfd3b13d1 |
(See sharp/lod-history) LOD, shadows, greedy meshing, new lighting, perf
--- Pretty much a Veloren fork at this point. Here's a high level overview of the changes (will be added to CHANGELOG just before merge). At a high level this MR incorporates roughly two groups of changes. The first group consists of new game features: more flexible map sizes, level of detail terrain, shadow maps, and a new lighting engine. This is "feature work" that (mostly) only adds new things to Veloren, and mostly shouldn't affect old stuff. The second big group of changes are those addressing the fallout from all the new features. These include performance fixes of various sorts: the addition of multiple graphics options and optimization of the cheap ones to avoid work, switching all voxel models to use some variant of greedy meshing, switching over much of our CPU-side vector math to exploit SIMD instructions (coinciding with a fork of `vek`), and a rewrite of how the UI handles text rendering (coinciding with updates to our fork of `conrod`). Making Veloren's hardcoded colors appear correct under the new lighting engine also required considerably changes (TODO: Fill in this section when it's complete). The second category of changes often heavily touches code owned by other people, including frequently modified code "owned" by a handful of people, so I recommend that this code be reviewed particularly carefully. --- At a high level (each will be described in more detail below): - The world map has been refactored. - The world size is no longer hardcoded (@zesterer). - The map generation code was made generic to allow using it outside of the `world` crate (@zesterer). - On world creation, we now compute *horizon maps* (@zesterer). - The way we pass the world from the server to the client has been updated (@xMAC94x). - Artifacts related to image rotation were fixed (@imbris). - Multiflow rivers were enabled (@zesterer). - In the process of making changes related to the world map, various incidental fixes and optimizations were required. - The new *level of detail* feature was added (@zesterer wrote part of this and has checked out the rest). - A new LOD terrain rendering step was added to the pipeline. - The LOD terrain quality was made configurable via a graphics setting. - Horizon maps were used to cast shadows from LOD chunks on both LOD and non-LOD terrain. - A "voxelization" effect was incorporated into rendered LOD terrain to make it blend better into the world. - In the process of making changes related to LOD, various incidental fixes and optimizations were required. - Veloren's lighting has been completely overhauled (@zesterer has already checked most of this out). - A semi-accurate index of refraction was assigned to our materials. - A new, more realistic, physically based approach to lighting was used using the *Ashikhmin Shirley* BRDF. - We emulate *atmospheric scattering* using equations designed for measuring solar panel light exposure. - We attempt to compute *realistic light attenuation* in water using its real material properties. - In the process of making changes related to LOD, various incidental fixes and optimizations were required. - Point and directional lights now cast realistic shadows, using *shadow mapping.* (@imbris, @zesterer, @Treeco, @YuriMomo) - Point light shadow maps were added to the rendering pipeline, using geometry shaders and *seamless cube maps*. - Directional light shadows were added to the rendering pipeline, using LISPSM together with disabling *depth clamping*. - "Shadow-only" chunks and NPCs were added to prevent shadows from models behind you from disappearing. - In the process of making changes related to shadow maps, various incidental fixes and optimizations were required. The addition of shadow maps, LOD terrain, and the new lighting all led to significant performance degradation, on top of other changes happening in master. Therefore, a large number of performance improvements were also needed: - The graphics options were made much more flexible and configurable, and shaders were optimize. - New options were provided for how to render lights and shadows (@Pfauenauge, @zesterer). - Graphic setting storage and configuration were overhauled to make adding new features easier (@Pfauenauge, @imbris). - Shaders were rewritten to utilize GLSL's preprocessor to avoid overhead (@zesterer, @YuriMomo). - In the process of making changes related to providing additional rendering options, various incidental fixes and optimizations were required. - Voxel model creation was switched to use *greedy meshing.* - A new voxel meshing method, greedy meshing, was added (@imbris). - Uses of the older meshing methods were migrated to use greedy meshing (@imbris, @jshipsey, @Pfauenauge). - New restrictions were added to terrain, figure, and sprites to future proof them for further optimizations (@jshipsey, @Pfauenauge, @zesterer). - Most positions are now relative to either chunk or player position for better precision (@imbris, @zesterer, @scottc). - In the process of making changes related to greedy meshing, various incidental fixes and optimizations were required. - Animation and terrain math were switched to use SIMD where possible. - Fixes were made to vek to make its SIMD feature usable for us (@zesterer, @imbris). - The interface and types used in bone animation were changed in various ways (@jshipsey, @Snowram, @Pfauenauge). - Redundant code generation for body animation is now partly taken care of by a macro (@jshipsey, @Snowram, @Pfauenauge). - Animation code was modified to to use vek's SIMD representation where possible (@jshipsey, @Snowram, @Pfauenauge). - Terrain meshing code and shadow map math were also modified to use vek's SIMD representation (@imbris). - SIMD instruction generation was enabled (@YuriMomo, @jshipsey, @Snowram, @imbris, @Angelonfira, @xMAC94x). - In the process of making changes related to greedy meshing, various incidental fixes and optimizations were required. - The way we cache glyphs was completely refactored, fixed, and optimized. - Our fork of `conrod` was optimized in various ways (@imbris). - Our fork of `conrod` now exposes whether a widget was updated during the current frame (@imbris). - Our use of the glyph cache was rewritten for correctness (@imbris). - A *text cache* was introduced that lets us skip remeshing glyphs that have not changed (@imbris). - Various changes were made to reduce pressure on the glyph cache, with more planned (@imbris, @Pfauenauge). - In the process of making changes related to the glyph cache, various incidental fixes and optimizations were required. - Colors were changed to keep Veloren's look consistent with master. - Some older tree models were brought back (@Pfauenauge). - TODO(@Sharp): All hardcoded colors were extracted and made hotloadable. - TODO(@Treeco, @Pfauenauge, @imbris, @jshipsey): Hardcoded colors were fixed to conform to Veloren's style. - TODO(@Treeco, @Pfauenauge, @imbris, @jshipsey): Color models were fixed to conform to Veloren's style. A detailed description of the involved changes follows. --- - The world size is no longer hardcoded. All functions dependent on world size now take a `WorldSizeLg`, which holds the base 2 logarithm of each actual world dimension and is guaranteed to maintain certain properties (outlined in `common/src/terrain/map.rs`). Additionally, many utility functions that utilize the world size were moved into `common` as well (mostly `common/src/terrain/mod.rs`). Finally, the world map format was updated in order to store its size explicitly, with a migration path from the old format that should work whenever the old formatted map was a square (practically always). See `world/src/sim/mod.rs` for these changes. - The map generation code was made generic to allow using it outside of the `world` crate. The parts of the map generating code that do not need to query the world were moved over to `common/src/terrain/map.rs`, allowing them to be used from the client without creating a dependency on `world`. The rest of it was turned into helper functions in `world/src/sim/map.rs`, which can be passed as closures to the generic map generation code to complete its construction. This also means that colors are now passed in separately to the map generation function. See <https://veloren.net/devblog-78/> for more details. - On world creation, we now compute *horizon maps*. See the function in `world/src/sim/util.rs`. Given a height map and a plane intersecting that height map, our horizon maps allow us to encode enough information to reconstruct shadows for each point on the height map using only the *horizon angle* (the angle at which the sun starts to become visible). As Veloren's sun only covers one plane, this is sufficient for encoding sun shadows for LOD terrain, by encoding two angles per chunk (one for each 90 degrees the sun covers). We can also use this for the moon, if we want, since the moon follows the same path. Additionally, we store the *height* of the furthest occluder, to try to make the shadows volumetric; so this means 4 bytes in total for each chunk. Support for horizon maps has been merged into the map functionality in common as well. - The way we pass the world from the server to the client has been updated. Rather than passing the prerendered map, we instead pass three maps with values for each chunk; one with the color information, a second with altitude information, and a third with horizon map information. We then reconstruct the map on the client, together with some additional information we send from the server (like the sea level and maximum height). See `common/src/msg/server.rs` for a detailed description of the format of `WorldMapMsg`, and `server/src/libr.rs` and `client/src/lib.rs` for details of the map construction and parsing. - Artifacts related to image rotation were fixed. See the commit message for commit SHA `cf74d55f2e3d2ae7d25fd68d5c73b01a6afde86e` for a detailed explanation. This involved changes to shaders, the addition of a new type of graphic (also reflected in the graphic cache) that allows specifying a border color (which automatically makes the associated texture immutable), and some related fixes. I reproduce the first two paragraphs of the MR description as well: ``` Fix map image artifacts and remove unneeded allocations. Specifically, we address three concerns (the image stretching during rotation, artifacts around the image due to clamping to the nearest border color when the image is drawn to a larger space than the image itself takes up, and potential artifacts around a rotated image which accidentally ended up in an atlas and didn't have enough extra space to guarantee the rotation would work). ``` - Multiflow rivers were enabled. This does not really need to be part of this MR, and would be easy to revert, but since it seemed to provide a nice improvement it's currently packaged with it. We already computed multiple outflows from each chunk for erosion purposes long before this MR. However, we never modified river rendering to be able to handle this case (just a single downhill river flow is complex enough!) so this was not exposed when deciding which chunks were rivers. Now that - In the process of making changes related to the world map, various incidental fixes and optimizations were required. Some examples of fixes include making sure terrain is never lowered to below sea level (to make the shadow maps report correct values), fixing map altitudes and colors to understand things like cliffs and "block level" coloring (that doesn'te xist on the column level), and fixing a crashbug when rendering images for the UI where source pixels are strongly rectangular. Some examples of related performance fixes include avoiding allocating a fresh vector for all the maps (i.e. copying it over to change the format from `[u32; n]` to `DynamicImage` and then copying again to convert to `RgbaImage`), and instead using the `gfx::memory::slice` function to accomplish the same thing. These sorts of changes are spread all arond the code. This includes the additon of a new scene, `voxygen/src/scene/lod.rs`, a new pipeline `voxygen/src/render/pipeline/lod_terrain.rs`, and new shaders `assets/vxygen/shaders/lod-terrain-vert.glsl` and `assets/vxygen/shaders/lod-terrain-frag.glsl`, as well as associated changes to the renderer in `voxygen/src/render/renderer.rs`. The main idea behind our initial approach to LOD was to take the world data we now get from the server (altitude, color, and horizon mapping). - Some previously computed values were turned into shader uniforms for better prediction on weak processors. (@zesterer) - Calls to power or trig functions were removed or replaced with multiplications, where possible. - After some deliberation - To properly handle sprite "waving" for nearby sprites, We explicitly designed the greedy meshing system with figures and sprites in mind. In both cases, we want to be able to *efficiently* pack many different models into the same texture, especially in cases where we know we will either not be removing any of the grouped-together from the models from the texture, or will remove all of them at once (so they can be packed into some specific subtexture). For sprites, since we know every model in advance and never intend to deallocate them, we currently pack them all as efficiently as possible into one giant tetxure atlas. However, in the future we might opt to pack them slightly less efficiently in exchange for shrinking the sprite vertex size. For figures, we pack all the textures for each *model* into the same atlas. is a global texture atlas used for every sprite, and for figures which is why we have the ability to mesh multiple models to the same texture area (using the simple texture atlas allocator) without requiring intermediate vector allocations. This is accomplished by delaying the time when we actually write the color and light data to the texture until *after* all the model vertices have been meshed; then, we can just allocate the whole color/light array at once, making the atlas we use an exact fit. In computer science-y terms, we accomplish this delay by, after we perform the initial greedy meshing (without texture information), not continuing to create the texture data, but instead constructing a *continuation*--that is, a function that, when called, will execute the rest of the computation. We push this continuation (which in Rust terms is a `FnOnce` closure that takes the `ColLightsInfo` that it is supposed to write to as context) onto a onto a vector resizing. To allow for suspended writes to texture data, Rust pointed out to me that the continuation that would eventually write the color and light data to the texture atlas (the one that is shared by all models sharing the same greedy mesher) would have to *own* whatever data it mshed. Because we often generate the model data to mesh as a temporary in `voxygen/src/load.rs`, the - Matrix multiplications in the shader were reduced for figure data (@zesterer). - Vertex "waves" for fluid data were removed. - Terrain "bending" near edges was removed. - Scaling was fixed to make sure empty space was not introduced in a space previously occupied by a block. It was also changed to take ownership of its voxel data, rather than sharing it, to let it be used with meshing. - Rust's nightly version was bumped in order to use the `array_map` function, which lets us reuse more code between the simple map and `FigureModelCache`. - PositionedGlyph::standalone. --- I tried to cite sources in many cases[^realtime],[^lloyd],[^lispsm],[^pbrt],[^greedy],[^tjunctions] where I needed features from elsewhere but I am particularly grateful for the following resources, esepcially where they have accompanying source code. I linked all of them that are accessible to the public (those that are not were obtained through legal means). [^realtime]: Eisemann, Elmar, Michael Schwarz, Ulf Assarsson, Michael Wimmer. Real-Time Shadows. A K Peters/CRC Press (T&F), 20160419. [^lloyd]: Lloyd,B. 2007. [Logarithmic perspective shadow maps](http://gamma.cs.unc.edu/papers/documents/dissertations/lloyd07.pdf). PhD thesis, University of North Carolina. [^lispsm]: Wimmer, M., Scherzer, D., and Purgathofer, W. 2004. [Light space perspective shadow maps](http://gamma.cs.unc.edu/papers/documents/dissertations/lloyd07.pdf). In Proceedings of Eurographics Symposium on Rendering 2004, pp. 143– 152. [^pbrt]: Pharr, Matt, et al. [http://gamma.cs.unc.edu/papers/documents/dissertations/lloyd07.pdf](Physically Based Rendering: From Theory to Implementation). Third edition, Morgan Kaufmann Publishers/Elsevier, 2017. [^greedy]: mikolalysenko. “Meshing in a Minecraft Game.” 0 FPS, 30 June 2012, <https://0fps.net/2012/06/30/meshing-in-a-minecraft-game/>. [^tjunctions]: blackflux. “Meshing in Voxel Engines – Part 1.” Blackflux.Com, 23 Feb. 2014, <https://blackflux.wordpress.com/2014/02/23/meshing-in-voxel-engines-part-1/>. I am also especially grateful to Khronos, Wikiepdia, and stackoverflow for answering many of my specific questions while writing the MR. --- Squashed commit of the following: commit |
||
|
Marcel Märtens
|
8687740265 | fix clippy warnings in new version | ||
|
Marcel Märtens
|
10970841cc |
fix master / update toolchain to 2020-08-15
|
||
|
Marcel Märtens
|
6db9c6f91b |
fix a followup bug, after a protocol fail now Participant is closed, including all streams, so we get the stream errors.
We MUST handle them and we are not allowed to act on a stream after it failed, as i am to lazy to change the structure to ensure the client to be imeadiatly dropped i added a AtomicBool to it. |
||
|
Marcel Märtens
|
9d32e3f884 |
proper voxygen connect and code cleanups:
- voxygen abort when the server has a invalid veloren_network handshake, e.g. by outdated version instead of try again - rename Network `Address` to `ProtocolAddr` as sugested by zest as it's a combination of Protocol and std::io::Addr - remove the manual byte arrays in `protocols.rs` with something more nice |
||
|
Marcel Märtens
|
8fb445b0e8 | better lz4 | ||
|
Marcel Märtens
|
1da6f15a43 | fixing various smaller network party from issue 657 | ||
|
Marcel Märtens
|
e1b27c51f5 | fix clippy issues in tests and add it to CI | ||
|
Marcel Märtens
|
2e3d5f87db |
StreamError::Deserialize is now triggered when recv fails because of wrong type
- added PartialEq to StreamError for test purposes (only yet!)
- removed async_recv example as it's no longer for any use.
It was created before the COMPLETE REWRITE in order to verify that my own async interface on top of mio works.
However it's now guaranteed by async-std and futures. no need for a special test
- remove uvth from dependencies and replace it with a `FnOnce`
- fix ALL clippy (network) lints
- basic fix for a channel drop scenario:
TODO: this needs some further fixes
up to know only destruction of participant by api was covered correctly.
we had an issue when the underlying channels got dropped. So now we have a participant without channels.
We need to buffer the requests and try to reopen a channel ASAP!
If no channel could be reopened we need to close the Participant, while
a) leaving the BParticipant in takt, knowing that it only waits for a propper close by scheduler
b) close the BParticipant gracefully. Notifying the scheduler to remove its stuff (either scheduler schould detect a stopped BParticipant or BParticipant will send Scheduler it's own destruction, and then Scheduler just does the same like when API forces a close)
Keep the Participant alive and wait for the api to acces BParticipant to notice it's closed and then wait for a disconnect which isn't doing anything as it was already cleaned up in the background
|
||
|
Marcel Märtens
|
3324c08640 |
Fixing the DEADLOCK in handshake -> channel creation
- this bug was initially called imbris bug, as it happened on his runners and i couldn't reproduce it locally at fist :)
- When in a Handshake a seperate mpsc::Channel was created for (Cid, Frame) transport
however the protocol could already catch non handshake data any more and push in into this
mpsc::Channel.
Then this channel got dropped and a fresh one was created for the network::Channel.
These droped Frames are ofc a BUG!
I tried multiple things to solve this:
- dont create a new mpsc::Channel, but instead bind it to the Protocol itself and always use 1.
This would work theoretically, but in bParticipant side we are using 1 mpsc::Channel<(Cid, Frame)>
to handle ALL the network::channel.
If now ever Protocol would have it's own, and with that every network::Channel had it's own it would no longer work out
Bad Idea...
- using the first method but creating the mpsc::Channel inside the scheduler instead protocol neither works, as the
scheduler doesnt know the remote_pid yet
- i dont want a hack to say the protocol only listen to 2 messages and then stop no matter what
So i switched over to the simply method now:
- Do everything like before with 2 mpsc::Channels
- after the handshake. close the receiver and listen for all remaining (cid, frame) combinations
- when starting the channel, reapply them to the new sender/listener combination
- added tracing
- switched Protocol RwLock to Mutex, as it's only ever 1
- Additionally changed the layout and introduces the c2w_frame_s and w2s_cid_frame_s name schema
- Fixed a bug in scheduler which WOULD cause a DEADLOCK if handshake would fail
- fixd a but in api_send_send_main, i need to store the stream_p otherwise it's immeadiatly closed and a stream_a.send() isn't guaranteed
- add extra test to verify that a send message is received even if the Stream is already closed
- changed OutGoing to Outgoing
- fixed a bug that `metrics.tick()` was never called
- removed 2 unused nightly features and added `deny_code`
|
||
|
Marcel Märtens
|
007f5cabaa | DOCUMENTATION for everything | ||
|
Marcel Märtens
|
661060808d |
switch from serde to manually for speed, remove async_serde
- removing async_serde as it seems to be not usefull the idea was because deserialising is slow parallising it could speed up. Whoever we need to keep the order of frames, (at least for controlframes) so serialising in threads would be quite complicated. Also serialisation is quite fast, about 1 Gbit/s such speed is enough for messaging, it's more important to serve parallel streams better. Thats why i am removing async serde coding for now - frames are no longer serialized by serde, by byte by byte manually, increadible speed upgrade - more metrics - switch channel_creator into for_each_concurrent - removing some pool.spwan_ok() as they dont allow me to use self - reduce features needed |
||
|
Marcel Märtens
|
2ee18b1fd8 |
Examples, HUGE fixes, test, make it alot smother
- switch `listen` to async in oder to verify if the bind was successful - Introduce the following examples - network speed - chat - fileshare - add additional tests - fix dropping stream before last messages can be handled bug, when dropping a stream, BParticipant will wait for prio to be empty before dropping the stream and sending the signal - correct closing of stream and participant - move tcp to protocols and create udp front and backend - tracing and fixing a bug that is caused by not waiting for configuration after receiving a frame - fix a bug in network-speed, but there is still a bug if trace=warn after 2.000.000 messages the server doesnt get that client has shut down and seems to lock somewhere. hard to reproduce open tasks [ ] verify UDP works correctly, especcially the connect! [ ] implements UDP shutdown correctly, the one created in connect! [ ] unify logging [ ] fill metrics [ ] fix dropping stream before last messages can be handled bug [ ] add documentation [ ] add benchmarks [ ] remove async_serde??? [ ] add mpsc |
||
|
Marcel Märtens
|
595f1502b3 |
COMPLETE REWRITE
- use async_std and implement a async serialisaition - new participant, stream and drop on the participant - sending and receiving on streams |
||
|
Marcel Märtens
|
a6f1e3f176 | Add a speedtest program to benchmark networking | ||
|
Marcel Märtens
|
10863eed14 | remove worker folder - flatten file structure | ||
|
Marcel Märtens
|
5c5b33bd2a |
Bring networking tests to green
- Seperate worker into own directory - implement correct handshakes - implement correct receiving |
||
|
Marcel Märtens
|
52078f2251 |
first implementation of connect and tcp using a mio worker protocol and:
- introduce a loadtest, for tcp messages - cleanup api - added a unittest - prepared a handshake message, which will in next commits get removed again - experimental mio worker merges - using uuid for participant id |
||
|
Marcel Märtens
|
a01afd0c86 | initial implementation of a network api |