Hell Let Loose News

[p]Hello all,[/p][p]First off, we want to thank everyone who has stuck with us through Update 18. We understand how frustrating the latest update has been, and we’re committed to making sure something like this doesn’t happen again.[/p][p]With that in mind, we’re looking ahead to upcoming content and would love to get you, the community, more involved in the direction we’re heading.[/p][p]This dev brief includes:[/p]

• [p]Update on connectivity issues and disconnecting issues[/p]
• [p]Information on this week’s community playtest[/p]
• [p]What is Smolensk?[/p]

[p]First up however, we have seen your comments on connectivity issues in game, and wanted to address them head-on. [/p][p][/p][p]Over the weekend, one of our major server providers, Qonzer, confirmed that a large-scale 29.1TB/s DDoS attack has affected one of it's core protection networks. Although only 15-20% of clients were affected some of those have been Hell Let Loose servers. The attack, which for Qonzer / NEOProtect reached 29.1 terabits powered by the Aisuru Botnet, has caused widespread disruption across multiple large platforms with the majority being gaming related.[/p][p]Recently attacks, which 1 year ago would reach between 10-30GB/s, are now unleashing anywhere between 100-1000GB/s and in some cases even higher.[/p][p]While we know parts of this botnet and others like it are able to be rented in a pay-to-DDoS platform, they seem to be pointed towards some of the top listed HLL servers after they've been seeded. This is not an issue within Team17 or the game itself, but a result of server providers coming under attack.[/p][p]We’re working closely with Qonzer as they investigate and implement their new protections with partners NEOProtect. Their upcoming statement, expected tomorrow, will outline further details, including a move to new server IP addresses to help stabilise performance.[/p][p]From our side, Team17 is exploring ways to adjust the in-game favouriting system so that it isn’t tied to specific IP addresses as well as offering support to the server providers where we can.[/p][p][/p][p]We’ll share more information tomorrow. Thank you for your patience and continued support while we work alongside our partners and cyber-security experts to restore stability.[/p][p][/p][p]We also wanted to share that Hotfix 3 will be dropping very soon, fixing a number of client-side issues affecting players. More information to follow soon.[/p][p] [/p][p]Now, time to enter the Battle for Smolensk.[/p][p][/p][p][/p][p][/p][p]This Friday, we’ll be holding our first Public Testing Environment (PTE) for our next map.[/p][p]We heard your feedback after Update 18, and going forward, we’ll be working more closely with the community before updates go live: whether it’s for new maps, vehicles, or major gameplay changes.[/p][p]This starts with our first PTE for the brand-new Smolensk map. As before, this will be a Steam-only event and will require you to download the Hell Let Loose (Public Testing) application.[/p][p]During this test, we’ll have several servers open, including dedicated servers for competitive, clan, and community-led matches.[/p][p]If you’re part of a clan and would like to get involved, please reach out to your server owner or contact our Community Manager, Ben B.[/p][p][/p][h2]Smolensk[/h2][p]Smolensk is one of our most ambitious maps yet. With a focus on urban, multi-story combat, this city-heavy battlefield is the largest urban map in the game so far. It features three distinct regions, industrial, urban, and riverside, each offering new tactical challenges and gameplay opportunities.[/p][p]During the PTE, please report any issues you encounter, such as invisible walls, bugged debris, or other map-related problems.[/p][p][/p][p][/p][p][/p][h2]SPAs[/h2][p]Next up, we’re introducing Self-Propelled Artillery (SPAs), the newest addition to the battlefield.[/p][p]These vehicles are part of our ongoing artillery rework and are expected to bring major gameplay changes. Because of this, we’ll be looking for as much community feedback as possible—both positive and negative.[/p][p]Your feedback on these systems will be vital for the development team moving forward.[/p][p][/p][p][/p][p][/p][p]The PTE will be going live on Friday 7th November. During the course of the PTE, a number of our incredible communities will be joining the server for games to ensure that we have full lobbies.[/p][p]These include (but aren’t limited to) communities such as Glow’s Battlegrounds hosting some community games, DraftHaus, PF and For Tea & Country joining the action, and a number of competitive matches will be getting played including some hard fought FNF games![/p][p]If you are wanting to get involved, be sure to speak to your server owner, or community lead to see whether they have any activities planned for the PTE.[/p][p][/p][p]For those who haven’t had a chance to read our Smolensk announcement blog, here’s a brief overview of our next map.[/p][p]Smolensk will be the newest addition to the battle on the Eastern Front, created in direct response to community requests for more urban Eastern Front battlefields. Set on the banks of the Dnieper River in the summer of 1943, this version of Smolensk captures a pivotal moment in the Suvorov Operation, when Soviet forces launched a major offensive to retake the city from German control after losing it in 1941 during the early stages of Operation Barbarossa.[/p][p]This new map will also be Hell Let Loose’s largest urban battlefield to date, featuring unique design elements that reshape how engagements unfold. Expect a scale and intensity unlike anything we’ve released before.[/p][p]As you fight for control of the city, you’ll notice how the train tracks and the Dnieper River both cut vertically through the map rather than across it, creating new lanes of movement and forcing teams to think carefully about positioning and flanking. Meanwhile, the addition of three-story buildings, fortified walls, and an overpass brings vertical combat to the forefront. Players will need to look up as often as they look ahead, with enemies able to take positions in ways never before seen in Hell Let Loose.[/p][p]For a deeper dive into the map’s development and features, head over to Dev Brief #208 to learn more about what awaits you when you enter Smolensk this Friday.[/p][p][/p][p][/p][p][/p]

[p][/p][p]We’re pleased to confirm that Hotfix 2 for Update 18 is now live across all platforms.[/p][p][/p][p]This update contains the fix for the crashing issues that have affected Hell Let Loose since the release of Update 18. The fix makes both the client and server more robust against assets with incorrect LOD materials that could previously cause the physics engine to crash. As a result, both servers and clients will now be much more stable than before.[/p][p][/p][p]We want to thank everyone who shared crash reports, feedback, and support during our investigation and testing over the past few weeks. Your patience and cooperation have been invaluable in helping us deliver this fix.[/p][p][/p][p]Alongside the release of Hotfix 2, the team continues to monitor live performance and community feedback closely. We will share further updates in the coming weeks, including our plans for Update 19 and beyond.[/p][p][/p][p]We’re committed to ensuring Hell Let Loose delivers the most stable and enjoyable experience possible, and we couldn’t have done this without your ongoing support.[/p][p][/p][p]Thank you,
The Hell Let Loose Team[/p]

[p][/p][p]We’re pleased to announce that Hotfix 2 for Update 18 will release tomorrow for all platforms.[/p][p][/p][p]We understand that the past few weeks since Update 18 have been challenging and we recognise how much this has impacted the community. We take full responsibility for this and want to reassure you that we’ve taken important lessons away from the experience.[/p][p][/p][p]Over the coming weeks, we’ll be sharing our plans to strengthen communication and feedback opportunities with the community. If you have any suggestions of things you’d like to see, be sure to let us know in the comments below.[/p][p][/p][p]We’ve also listened closely to your feedback and have made the decision to delay the release of Update 19. This will give us additional time to ensure that it launches in the best possible state. We’ll share the new release date soon.[/p][p][/p][p]We’d like to extend a special thank-you to server owners, community testers, and players who supported the team throughout the investigation and testing of Hotfix 2. Your cooperation and feedback were instrumental in helping us identify the issue and verify the fix.[/p][p][/p][p]Following the rollout of Hotfix 2, we’ll continue to post regular updates as we move toward the release of Update 19. For those who want to see the technical details, please be sure to read the blog post from Friday 24 October where Kieran, Studio Technical Director at Expression Games, goes into detail on the reason behind the crash and what’s been done to resolve this.[/p][p][/p][p]We remain fully committed to making Hell Let Loose the best experience it can be for all players, and we’re excited to share more about Update 19 and our plans for 2026 in the near future.[/p][p][/p][p]Thank you again for your patience and support over the past few weeks. Your feedback continues to shape the future of Hell Let Loose, and we’re grateful for everyone who has shared their thoughts with us. We’re looking forward to finishing 2025 strong with you, including the release of the brand-new Smolensk map.[/p]

[p][/p][p]Today marks a major milestone in our ongoing efforts to resolve the stability and crashing issues that have affected Hell Let Loose since Update 18.[/p][p]After many days of investigation, testing, and community support, we’re pleased to share that the team has successfully identified the root cause of the problem and implemented a fix. This new build has been tested and is almost ready for an rollout as Hotfix 2.[/p][p]We know how disruptive the last week has been for many of you, and we want to thank everyone who has continued to provide reports, feedback, and patience as we worked to get the game back to a stable state. Your input has been instrumental in helping the team reach this point.[/p][p][/p][h3]TLDR Summary[/h3]

• [p]The team has identified the root cause of the crashes affecting Hell Let Loose since Update 18.[/p]
• [p]The issue was linked to a physics material reference error caused by certain assets, which could result in client and server crashes.[/p]
• [p]A new build containing a fix for both client and server crashes has been created, tested, and certified.[/p]
• [p]This fix is now available for Steam players and server owners through a new ‘Experimental Branch’.[/p]
• [p]The full Hotfix 2 release will roll out on Steam and other platforms early next week following platform approvals.[/p]
• [p]Players and server hosts on the main branch or console platforms are advised to remove the following maps from rotation over the weekend:
  Carentan, Foy, Omaha, Utah, Remagen, Sainte-Mère-Église (SME), Stalingrad[/p]
• [p]The team will continue to monitor for any remaining issues, while also preparing additional asset corrections for Update 19.[/p]
• [p]Thank you to everyone who provided reports, feedback, and patience. Your support directly contributed to identifying and fixing the issue.[/p]

[p][/p][h2]Status of the Investigation[/h2][p][/p][h3]Latest Update[/h3][p]After yesterday’s breakthrough in reproducing the crash, Kieran and his team worked late into the night to pinpoint the exact cause. It wasn’t an easy process, but by the end of the evening they had successfully identified the specific issue, allowing work on the fix to begin immediately.[/p][p]This morning, the first new build containing the fix was completed. It also includes an additional correction for another potential fault in the same chain of logic.[/p][p]Since then, our QA team has been rigorously testing the new build, which was successfully certified this afternoon.[/p][p]With this update, we have addressed both client crashes (often seen as mass disconnects) and server crashes (which previously caused players to lose connection).[/p][p]We would like to note that, due to the nature of the issue, there may have been other undefined behaviours linked to it that did not always result in a crash. These should now also be resolved. However, following this hotfix, we will continue monitoring to ensure there are no remaining causes of client disconnections.[/p][p][/p][h3]Details on the Hotfix 2 for Update 18[/h3][p]Thank you all for your patience. We can now confirm that a new build of Hell Let Loose has been created and includes a fix for the crashing issue. Below are the key details to be aware of:[/p][p][/p][h3]1. New Steam Branch:[/h3][p]A new Experimental Branch is now available on Steam. You can access it by navigating to:
Steam → Hell Let Loose → Properties → Betas → Experimental Branch[/p][p][/p][p][/p][p]Head to the Properties section of the Hell Let Loose app on Steam. Do this by right clicking on Hell Let Loose, or hitting the settings cog within the Steam page.[/p][p] [/p][p][/p][p]Go to Betas, and then select ‘experimental_branch’[/p][p] [/p]

• [p]The “None” option is the default/main branch that has the current Live 18.0 Hotfix 1 build[/p]
• [p]The “experimental_branch” right now has 18.0 Hotfix 2 build[/p]

[p][/p][p]Switching to the new branch in the Steam client is a 3.2GB download. Those that switch, once the patch rolls out globally next week, will need to switch back to the main branch ‘None’. Once back to the default branch, you will be instantly in as it will be the same build at that point.[/p][p]This branch includes the client-side fix, and community servers can now opt to move across. A number of official servers are also live on this build.[/p][p]Please note that this is an opt-in process. Servers and players using the Experimental Branch will not appear in the default server list. This setup is intended for those who want to play on the new build servers and on the new build.[/p][p]A full release of Hotfix 2 will come out on Steam early next week alongside the other platforms.[/p][p] [/p][h3]2. Other Platforms:[/h3][p]Due to approval requirements, this version will not go live on other platforms today. It is currently scheduled for release early next week.[/p][p]In the meantime, for all other platforms (and for those on Steam who have already begun seeding on the main branch) we have shared a list of maps that have been identified as potential triggers for the crash:[/p]

• [p]Carentan[/p]
• [p]Foy[/p]
• [p]Omaha[/p]
• [p]Utah[/p]
• [p]Remagen[/p]
• [p]Sainte-Mère-Église (SME)[/p]
• [p]Stalingrad[/p]

[p]These maps have the highest likelihood of triggering the crash. We strongly recommend removing them from rotation if you are on the main Steam branch or any other platform (Epic, WinStore, Xbox, or PS5).[/p][p]Thank you again for your continued patience and support. It has been invaluable throughout this process, and we deeply appreciate everyone in the community for working with us as we’ve resolved the issue.[/p][p][/p][h2]From the Dev Team: Understanding the Issue[/h2][p]Yesterday, Kieran, Studio Technical Director at Expression Games, shared a detailed breakdown of the work happening behind the scenes to uncover the root cause of the crashes, and why this issue only appeared in the live environment. Following the breakthrough we had last night, we were able to identify the cause of the crashing issue and deliver a fix.[/p][p]With this in mind, we wanted to continue our commitment to transparency by providing a deeper explanation of what went wrong and how it was resolved.[/p][p][/p][h3]How The Crash Presents Itself[/h3][p]The crash can manifest on any client or the game server. In both cases, one of the following functions would typically appear near the top of the call stack:[/p][p][/p][p]The crash appeared to happen randomly and didn’t have a 100% reproduction rate, which made it extremely difficult to track down.[/p][p][/p][h3]How The Core Issue Was Discovered[/h3][p]When the issue first began to appear in community reports post-Update 18, the team reviewed crashes captured through Backtrace in the development environment. We later confirmed that the issue had existed since 28 July 2025, with only four crashes recorded during that time (between 28 July - 16 October): two server-side and two client-side. This immediately showed how hard it was to reproduce under normal development conditions.[/p][p]At first, it was unclear whether the cause was related to server type (GPortal versus community servers), differences in server tools like CRCON, whether it required a full server populations, or specific gameplay situations that you you seen in a public game environment.[/p][p]To narrow things down, we explored all these possibilities in parallel. We collaborated closely with the community, server owners, and War Correspondents to gather reports and eliminate potential causes.[/p][p]Because the issue occurred so rarely during testing, it became clear that the only way to properly diagnose it was to continue investigating while the bug remained live in the game. If we had rolled back Update 18, there was a real chance we might never reproduce it during internal testing, which would have prevented us from finding and fixing the issue.[/p][p]On Wednesday 22 October, the first successful reproduction of the crash occurred on an live official server, ruling out both the server provider and external tools (e.g. CRCON) as contributing factors. During this time, we identified that the Foy map had the highest rate of reproductions, thanks to reports from the server owners and War Correspondents.[/p][p]The next day (Thursday 23 October), QA focused on Foy specifically. At that point, there was a theory that the new artillery strike commander ability might be related. While that turned out to be true, although not in the way initially suspected, it was during these tests that we got lucky.[/p][p]A QA tester experienced a crash after entering a building. One by one, other testers entered the same building and also crashed. That gave us our 100% reproduction case.[/p][p][/p][h3]What The Problem Was[/h3][p]When performing physics tests in Hell Let Loose, the game often requests the physical material of the object being interacted with. This information determines what visual effects, particles, and sounds are used when bullets or projectiles hit that surface.[/p][p]With Update 18, some assets were re-imported to resolve texture issues, but this process unintentionally affected their Level of Detail (LOD) setup. As a result, certain objects were referencing one fewer material than they should have.[/p][p]When the physics system interacted with these assets, it sometimes attempted to read a material from an invalid memory address. Depending on what that memory address pointed to, what happens next is down to chance, with one of two outcomes occuring:[/p]

• [p]The game continued running, unknowingly using corrupted data.[/p]
• [p]The game crashed immediately in one of the previously listed functions.[/p]

[p]This randomness explained why reproducing the crash was so difficult; it depended on chance whether the invalid pointer caused an instant crash or not.[/p][p]For reference, the three most likely causes of the crash, in order, are:[/p]

1. [p]Simulating a bullet trajectory, and it hitting a “bad” object[/p]
2. [p]Simulating a projectile (e.g. artillery), and it hitting a “bad” object[/p]
3. [p]Walking on a “bad” object[/p]

[p][/p][h3]What Was The Fix[/h3][p]Using the PhysX API directly, the team implemented a validation step to ensure that when retrieving a material index, it matches a valid material within the asset. If the index does not map correctly, the system safely exits that process without returning a physical material, preventing the crash.[/p][p]In practice, this means that for “bad” assets, the correct visual or sound effect may occasionally not appear. However, since this issue has existed for months without being noticed, the impact will be minimal, and most importantly, the game and servers will no longer crash.[/p][p]Over the coming weeks, the team will continue identifying and repairing these affected assets to ensure they are corrected for Update 19.[/p][p][/p][h3]How Will We Ensure That This Specific Issue This Doesn’t Happen Again[/h3][p]In future non-shipping builds, whenever this issue is detected, a red on-screen message will appear that reads:[/p][p][/p][p]This system allows QA to easily identify and report any problematic assets. Since the message clearly shows the actor and component names, the art team can quickly locate and fix the issue.[/p][p]Because this error can occur through any kind of physics interaction (shooting, bombing, or walking on an affected surface) QA will naturally encounter and resolve these assets during standard gameplay testing.[/p][p] [/p]

[p]Update on our investigations into the server and stability issues since Update 18.[/p][p][/p][p]Thank you to everyone who has continued to support us and share crash reports over the past few days. Your feedback and detailed reports have played a huge role in helping us make real progress.[/p][p]Today, we’re pleased to share that we’ve taken a major step forward in our investigation. The team has been able to consistently reproduce the crash, which brings us much closer to identifying the exact cause. We know how important stability is to your experience, and we remain fully committed to resolving this issue as quickly as possible.[/p][p][/p][h2]TLDR Summary[/h2]

• [p]The team has made a major breakthrough in the investigation.[/p]
• [p]We have now found a consistent way to reproduce the crash, allowing us to focus on a much narrower area of the game’s systems.[/p]
• [p]This issue has been extremely difficult to track down, as reproducing it in a QA or development environment was nearly impossible until now.[/p]
• [p]With this progress, the team is closing in on the root cause and continuing to work around the clock toward a fix.[/p]
• [p]We will share more information before the weekend so stay tuned.[/p]

[p][/p][h2]Status of the Investigation[/h2][p][/p][h2]Latest Update[/h2][p]Today marks a significant step forward in resolving the current crashing issues.[/p][p]As of this afternoon, the team has discovered a consistent way to trigger a crash. This breakthrough allows us to focus on a much smaller area of the game’s systems to pinpoint the exact cause.[/p][p]This particular issue has been a needle in a haystack to uncover. The conditions required to reproduce it are so specific that encountering it within a QA or development environment is almost impossible. Finding a reliable way to make it happen marks a major turning point in our investigation.[/p][p]The team continues to work around the clock to determine the root issue, and have been focused on getting this resolved as soon as possible.[/p][p]We will be sharing another update tomorrow and hope to have more information available before the weekend.[/p][p][/p][h2]From the Dev Team: Understanding the Issue[/h2][p]As part of our update yesterday, we shared more behind-the-scenes details about the process of tracking down the cause of the recent crashes. The response from the community has been incredibly supportive, and it’s been great to see players taking such an interest in how we’re approaching our investigation. With that in mind, we wanted to expand a little further on how we got here and what we’ve been doing to resolve the issue.[/p][p][/p][p]Here is an explanation from Kieran, Studio Technical Director at Expression Games:[/p]

[p]“The release build for Update 18 was starting to be finalised from 10 September, over a month ago. This means the underlying issue responsible for the current crashes has likely existed since that point, and possibly even earlier within our development environment.[/p][p]Prior to yesterday afternoon (22 October), we had only observed a handful of unrelated crashes internally, and none provided indicators of the issue that has since appeared in the live environment.[/p][p]The challenge we face is that this specific crash occurs when something has already gone wrong before the actual crash point. As a result, the crash dumps we collect, both from our internal testing and from the community, show where the crash happens within PhysX but provide little insight into how the physics engine reached that unstable state.[/p][p]Our next step is to fully understand what conditions lead to the crash so we can reliably reproduce it. Once we can reproduce it consistently, we can verify a fix and roll it out confidently to everyone.[/p][p]The good news is that on Wednesday 22 October, we successfully triggered the crash on a live server outside of the development environment for the first time. This discovery allowed us to focus our search on what causes the crash. By attaching development tools to clients playing on live servers, we were then able to collect detailed debugging information whenever a crash occurs, helping us better understand its cause.[/p][p]Using this approach, as of this afternoon we have now been able to reproduce the crash within the development environment. This marks a major step forward, and we are closing in on the root cause.[/p][p]We would like to thank everyone who has been submitting crash logs and reports during this process. Your contributions have been an invaluable resource in our investigation. We’re confident that once we’ve confirmed the exact issue, we’ll be able to prepare and deliver a fix as soon as possible. Your support means a great deal to me and to the wider development team.”[/p]