FS22 - TestRunner Tool for Testing Mods

Heizer
GIANTS Software | Gameplay Programmer
Posts: 1239
Joined: Thu Jul 02, 2009 5:00 pm

FS22 - TestRunner Tool for Testing Mods

Post by Heizer »

Tool for testing Farming Simulator 22 Mods
As with FS19 we are happy to provide our Mod Testing Tool updated for FS22.

TestRunner_public.exe
  • checks a mod and shows various errors and violations of ModHub guidelines
  • provides test result as html and xml document
  • tests are structured in multiple modules/groups
  • tool is also used by our QA internally (with an extended feature-set)

Goals
  • speed up the process of testing mods by helping the modder to identify and fix problems (before submitting to ModHub)
  • detect various errors and provide hints how to fix them
  • ensure ModHub guidelines are met
  • improve performance and minimize overall mod file-size and memory usage

Key Features / What it does (selected examples)
  • check if the descVersion matches current patch level
  • check for xml parsing errors
  • warn about wrongly exported dds textures
  • warn about missing or obsolete files
  • show duplicate files (within the mod and in the base game)
  • provide information about shader problems, which need fixing for a console release
  • and much more, for detailed explanations see "Test Cases" section
  • all found problems will be provided in form of an HTML document and and xml file
    Example Result HTML

Requirements for using the tool
  • Farming Simulator 22 (min. version 1.8)
  • GIANTS Editor (min. Version 9.0.3)
    => the path to game installation needs to be correctly set within the editor
  • latest version of the Farming Simulator 22 TestRunner (0.8.8)

Disclaimer
  • the tool is still in beta status, some tests might show errors which are none (so called "false positives")
    particularly "unused files" reported by module "ObsoleteFiles"
  • please report any cases where you think the tool outputs errors which are false positives
    when making a bugreport please include testrunner log, the result files (xml and html) and if possible the mod itself (or its id on the ModHub), so we can reproduce the issue
    ideally use the "TestRunner" project on our public Bugtracker for reporting any bugs

Instructions / How to use
  1. drag and drop the directory of an unzipped mod onto TestRunner_public.exe
    • note: folder of the mod does not have to be in the same directory as the TestRunner
    • a command-line window will open displaying current progress
    • if something went wrong an error message is displayed giving details on the problem
      all outputs are also saved in the "TestRunner.log"
  2. if requirements are met the tool will start to analyze the mod
    • the command line will display some information on the current progress
      unless there is a "critical" message, this can be ignored
    • GIANTS Editor will be automatically opened and closed during the process
    • note: depending on the size of the mod and the speed of the hard drive the process may take several minutes
  3. when the test process has finished, the message "Finished execution" will be displayed inside the command line
    • a xml and html file will be generated in the same directory as the TestRunner
    • files are named after the directory name of the tested mod and contain the result of the test
      • html file only contains all the found errors (per module)
      • the xml contains all errors and more detailed (meta)information
  4. the generated html file will be automatically opened in your browser where you can check if errors were found in your mod
    • errors are outlined in red
    • explanations are in italics
    • instruction on how to correct errors are colored green


Test Cases and explanations

In the following the individual modules and their test cases are explained in detail
Work in Progress
  • xml-file well-formedness (only required by the game to a certain extent, but should be fulfilled nonetheless)
  • Module DXTCheck: ensure dds files use correct DXT formats, suggest conversion to DXT1 where possible
  • Module MipMapCheck: ensure textures for 3D objects have MipMaps, warn if texture has too low resolution
  • Module TextureCheck: ensure textures have a resolution of 2^n
  • Module I3DCheck: check reference integrity, obsolete lights/cameras, textureArray usage
  • Module ModDescCheck: check a set of ModHub guidelines (descVersion, version number format, changelog, localization, ...)
  • Module VehicleCheck: ensure specific vehicle xml elements are present (e.g. "washable") or absent (e.g. "dailyUpkeep" in vehicles)
  • Module SlotCheck: calculate estimated number of used slots for each store item as a reference for the modder
  • Module ObsoleteFiles: list obsolete source files not loaded by the game, list possibly unused textures, list duplicate files
  • Module EditorCheck: load all i3d files contained in the mod and check for errors
  • Module ShaderCheckEditor: warn if custom shaders are present in the mod nullifying console compatibility, raise errors if material configurations are used which are not available in the shader cache

Troubleshooting
  • problem: game installation can not be found
    • reason: tools checks default install location and reads location from registry, make sure you didn't move the game after installation
    • solution: run in cmd (see instructions below) with argument -g <pathToGameInstallation> to manually set the path
      Image
  • problem: editor installation can not be found
    • tools searches for installed Giants editors in giantsPackageRegistry xml located in appdata/local
    • solution: run in cmd (see instructions below) with argument -p $FS22_EDITOR$ <pathToEditorExe>
  • problem: program closes down by itself/crashes
    • post/send TestRunner.log, if possible send the tested mod in a private message
    • if possible, run the tool from command line and add "--verbose" as an argument and send us the log
      this way more information will be included in the log-file and help us with debugging process


Instructions for using with command-line (cmd)

if you need to make additional settings (e.g. game installation cannot be found automatically), you can run the tool from a commandline
  1. start a command-line or powershell window
    • Option A: click the address bar of the windows explorer window, type "cmd", hit enter
      this will open a command-line at the current directory of the explorer
    • Option B: hold shift and right click the background of the explorer window, select "Open Commandline window here" or "Open PowerShell window here"
      this will open a command-line/powershell at the current directory of the explorer
    • Option C: Press "Win" + "R", type "cmd" in the dialog, hit enter
      this will open a command line in your user profile
      => navigate to the location of the TestRunner_public.exe
  2. type "TestRunner_public.exe" ("./TestRunner_public.exe" for PowerShell)
  3. hit space and enter the path to your unzipped mod, e.g. "D:\mods\FS22_caseIHMagnum724Pro"
  4. Optional: add additional arguments if required/desired
    Image
  5. hit enter to start the test tool

If you find any bugs I suggest to report them on our public Bugtracker, as its more efficient to manage them there.
We are looking forward to your Feedback and hope the tool will support you creating mods and submitting them to ModHub.
Gruß Heizer
User avatar
MrMarcel5
GIANTS Software | QA Analyst
Posts: 122
Joined: Thu Jul 21, 2016 3:08 pm

Re: FS22 - TestRunner Tool for Testing Mods

Post by MrMarcel5 »

Errors regarding "parseError: ('texdiag analyze regex failed'..." are likely to occur with textures converted by our texture converter.
If this is the case, these can be ignored for now.
User avatar
ALiEN JiM
Posts: 36
Joined: Mon Aug 27, 2018 10:52 pm

Re: FS22 - TestRunner Tool for Testing Mods

Post by ALiEN JiM »

MrMarcel5 wrote: Fri Jan 14, 2022 5:51 pm Errors regarding "parseError: ('texdiag analyze regex failed'..." are likely to occur with textures converted by out texture converter.
If this is the case, these can be ignored for now.
parseError also from store icons created by FS Icon Generator
User avatar
strauts6
Posts: 297
Joined: Mon Nov 03, 2014 12:00 am

Re: FS22 - TestRunner Tool for Testing Mods

Post by strauts6 »

Is this a false positive error or is there a proper way to adding custom placeables to a map?

Code: Select all

<filenotfound referencedin="FS22_Dunalka/maps/mapUS/storeItems.xml">maps/placeables/waterFillTriggers.xml</filenotfound>
Test tool says that the xml files cant be found, but they are there and works perfectly in game.
Minecraftchest1
Posts: 14
Joined: Sun Sep 13, 2020 11:37 pm

Re: FS22 - TestRunner Tool for Testing Mods

Post by Minecraftchest1 »

Command line tools. My favorite. Time to decide if I want to run tests out of CI/CD or not.
martin)
Posts: 3
Joined: Mon Jan 31, 2022 2:44 pm

Re: FS22 - TestRunner Tool for Testing Mods

Post by martin) »

ERROR root - No game installation path set/found
CavemanTechGaming
Posts: 1
Joined: Tue Mar 15, 2022 2:57 am

Re: FS22 - TestRunner Tool for Testing Mods

Post by CavemanTechGaming »

Got this error today
ERROR root - 'NoneType' object has no attribute 'with_suffix'
AttributeError: 'NoneType' object has no attribute 'with_suffix'

What does this mean?
xludo_modding
Posts: 1
Joined: Wed Jun 22, 2022 6:06 pm

Re: FS22 - TestRunner Tool for Testing Mods

Post by xludo_modding »

hi i can't connect with the game to the testrunner ? Help
liberoaro
Posts: 4
Joined: Mon Jun 05, 2023 12:49 am

Re: FS22 - TestRunner Tool for Testing Mods

Post by liberoaro »

martin) wrote: Thu Feb 17, 2022 9:00 pm ERROR root - No game installation path set/found
ja mam ten sam problem
Jarkozpl
Posts: 1
Joined: Fri Dec 08, 2023 5:23 pm

Re: FS22 - TestRunner Tool for Testing Mods

Post by Jarkozpl »

How to download it?
doodoorolf
Posts: 22
Joined: Thu Sep 08, 2022 11:40 pm

Re: FS22 - TestRunner Tool for Testing Mods

Post by doodoorolf »

Hallo zusammen
habe den neuesten Testrunner heruntergeladen und wollte einen MOD prüfen.
Der Testrunner startet und bleibt nach kurzer Zeit stehen und zeigt keine Reaktion.
Es öffnet sich der GE, jedoch der Mod wird nicht geladen und getestet.

LOG anbei des Testrunner.
https://image.giants-software.com/index ... codes=true

Welche Schritte muss ich unternehmen um den Testrunner zu verwenden.

Danke
LG

NACHTRAG:
Nach mehreren Versuchen zeigt er mir diese Fehlermeldung im GE.

Started 1 threads for threadpool 'TerrainPatchOccluderManager unique worker'
Error: Can't load resource 'C:/Users/G_tze Rolf/AppData/Local/Temp/_MEI162842/additionals/tmp_14836_loadI3DFiles.lua'.
Check for updates (https://gdn.giants-software.com)

Ein Update habe ich durchgeführt, leider ohne Erfolg
Die erwähnte Datei bei ERROR gibt es nicht in dem Verzeichnis
CyTec502
Posts: 7
Joined: Wed Aug 30, 2023 11:05 pm

Re: FS22 - TestRunner Tool for Testing Mods

Post by CyTec502 »

It's a stuff program. Does not work. Nothing can be done with it.
User avatar
LS-Lara
Posts: 358
Joined: Sun Aug 04, 2019 4:57 pm

Re: FS22 - TestRunner Tool for Testing Mods

Post by LS-Lara »

CyTec502 wrote: Wed Feb 14, 2024 6:33 pm It's a stuff program. Does not work. Nothing can be done with it.
It does exactly what it is supposed to do - if operated properly. What is the actual issue you encounter under which operational circumstances ?
Der Sinn des Lebens ist:
29.61%

Mein Traktor:
Base:
HP Pavilion 690-03xx
Core i7-8700 @ 3.2GHz
NVIDIA GeForce GTX 1060 6GB
2x Benq GL2450H
Windows 10 Home 64bit
Custom:
2 x 16GB Corsair Vengeance LPX DDR4 C16 XMP 2.0
Samsung NVMe M.2 970 EVO Plus 500GB
Samsung SSD 860 EVO 1TB
Logitech G203 Prodigy
Logitech Wireless F710
CyTec502
Posts: 7
Joined: Wed Aug 30, 2023 11:05 pm

Re: FS22 - TestRunner Tool for Testing Mods

Post by CyTec502 »

LS-Lara wrote: Thu Feb 15, 2024 7:36 am
CyTec502 wrote: Wed Feb 14, 2024 6:33 pm It's a stuff program. Does not work. Nothing can be done with it.
It does exactly what it is supposed to do - if operated properly. What is the actual issue you encounter under which operational circumstances ?
It starts. Press a key. Then exit.
Post Reply