cosmoxi, it is unlikely that this is a change between betas. It's more likely that the memory leak just happend "right now". If you notice any action that dramatically increases memory usage and it return back after playback, please let me know. I'm pretty sure there is a memory leak, the problem is, noone knows when exactly does it happen. (it could be specific file, specific decoder configuration - i.e. some direct show filters, if not using VLC, specific order of actions (i.e. multiple transcoding sessions), etc.)
VLC is installed and I'm using hardware decoding and encoding. Playback was on recorded TV, mpeg2. The mpeg2 files had random problems due to low quality signal so perhaps something is buggy and freaks out with errors in source files. Still testing
edit: removed VLC just for testing and played back an mpeg2 recorded under poor signal to try to recreate the problem. it came to a point in the video where playback just stopped because something in the workflow couldn't handle the exception in the source. I looked on my desktop and could see the memory usage rising. Problem is once I ask ATV4 app to exit out of the video the memory isn't released even though the server CPU and GPU tasking stops.