FS19 - TestRunner Tool for Testing Mods

Heizer
GIANTS Software | Gameplay Programmer
Posts: 1239
Joined: Thu Jul 02, 2009 5:00 pm

FS19 - TestRunner Tool for Testing Mods

Post by Heizer »

Tool for testing Farming Simulator 19 Mods
After a lengthy development and alpha-phase we are glad that we can finally release our first version of our Mod Testing Tool to you.

TestRunner_public.exe
  • checks a mod and shows various errors and violations of ModHub guidelines
  • provides test result as html and xml document
  • tests are structured in multiple modules/groups
  • tool is also used by our QA internally (with an extended feature-set)

Goals
  • speed up the process of testing mods by helping the modder to identify and fix problems (before submitting to ModHub)
  • detect various errors and provide hints how to fix them
  • ensure ModHub guidelines are met
  • improve performance and minimize overall mod file-size and memory usage

Key Features / What it does (selected examples)
  • check if the descVersion matches current patch level
  • check for xml parsing errors
  • warn about wrongly exported dds textures
  • warn about missing or obsolete files
  • show duplicate files (within the mod and with the base game)
  • provide information about shader problems, which need fixing for a console release
  • and much more, for detailed explanations see "Test Cases" section
  • all found problems will be provided in form of an HTML document and and xml file
    Example Result HTML

Requirements for using the tool
  • Farming Simulator 19 (min. version 1.7)
  • GIANTS Editor (min. Version 8.2.2)
    => the path to game installation needs to be correctly set within the editor
  • latest version of the TestRunner (0.6.3)

Disclaimer
  • the tool is still in beta status, some tests might show errors which are none (so called "false positives")
    particularly "unused files" reported by module "ObsoleteFiles"
  • please report any cases where you think the tool outputs errors which are false positives
    when making a bugreport please include testrunner log, the result files (xml and html) and if possible the mod itself (or its id on the ModHub), so we can reproduce the issue
    ideally use the "TestRunner" project on our public Bugtracker for reporting any bugs

Instructions / How to use
  1. drag and drop the directory of an unzipped mod onto TestRunner_public.exe
    • note: folder of the mod does not have to be in the same directory as the TestRunner
    • a command-line window will open displaying current progress
    • if something went wrong an error message is displayed giving details on the problem
      all outputs are also saved in the "TestRunner.log"
  2. if requirements are met the tool will start to analyze the mod
    • the command line will display some information on the current progress
      unless there is a "critical" message, this can be ignored
    • GIANTS Editor will be automatically opened and closed during the process
    • note: depending on the size of the mod and the speed of the hard drive the process may take several minutes
  3. when the test process has finished, the message "Finished execution" will be displayed inside the command line
    • a xml and html file will be generated in the same directory as the TestRunner
    • files are named after the directory name of the tested mod and contain the result of the test
      • html file only contains all the found errors (per module)
      • the xml contains all errors and more detailed (meta)information
  4. the generated html file will be automatically opened in your browser where you can check if errors were found in your mod
    • errors are outlined in red
    • explanations are in italics
    • instruction on how to correct errors are colored green


Test Cases and explainations

In the following the individual modules and their test cases are explained in detail
Work in Progress
  • xml-file well-formedness (only required by the game to a certain extent, but should be fulfilled nonetheless)
  • Module DXTCheck: ensure dds files use correct DXT formats, suggest conversion to DXT1 where possible
  • Module MipMapCheck: ensure textures for 3D objects have MipMaps, warn if UI overlay textures have MipMaps, warn if texture has too low resolution
  • Module TextureCheck: ensure textures have a resolution of 2^n
  • Module I3DCheck: check reference integrity, obsolete lights/cameras, textureArray usage
  • Module ModDescCheck: check a set of ModHub guidelines (descVersion, version number format, changelog, localization, ...)
  • Module VehicleCheck: ensure specific vehicle xml elements are present (e.g. "washable") or absent (e.g. "dailyUpkeep" in vehicles)
  • Module SlotCheck: calculate estimated number of used slots for each store item as a reference for the modder
  • Module ObsoleteFiles: list obsolete source files not loaded by the game, list possibly unused textures, list duplicate files
  • Module EditorCheck: load all i3d files contained in the mod and check for errors
  • Module ShaderCheckEditor: warn if custom shaders are present in the mod nullifying console compatibility, raise errors if material configurations are used which are not available in the shader cache

Troubleshooting
  • problem: game installation can not be found
    • reason: tools checks default install location and reads location from registry, make sure you didn't move the game after installation
    • solution: run in cmd (see instructions below) with argument -g <pathToGameInstallation> to manually set the path
      Image
  • problem: editor installation can not be found
    • tools searches for installed Giants editors in giantsPackageRegistry xml located in appdata/local
    • solution: run in cmd (see instructions below) with argument -p $FS19_EDITOR$ <pathToEditorExe>
  • problem: program closes down by itself/crashes
    • post/send TestRunner.log, if possible send the tested mod in a private message
    • if possible, run the tool from command line and add "--verbose" as an argument and send us the log
      this way more information will be included in the log-file and help us with debugging process


Instructions for using with command-line (cmd)

if you need to make additional settings (e.g. game installation cannot be found), you can run the tool from a commandline
  1. start a command-line or powershell window
    • Option A: click the address bar of the windows explorer window, type "cmd", hit enter
      this will open a command-line at the current directory of the explorer
    • Option B: hold shift and right click the background of the explorer window, select "Open Commandline window here" or "Open PowerShell window here"
      this will open a command-line/powershell at the current directory of the explorer
    • Option C: Press "Win" + "R", type "cmd" in the dialog, hit enter
      this will open a command line in your user profile
      => navigate to the location of the TestRunner_public.exe
  2. type "TestRunner_public.exe" ("./TestRunner_public.exe" for PowerShell)
  3. hit space and enter the path to your unzipped mod, e.g. "E:\mods\FS19_kirovetsK700A"
  4. Optional: add additional arguments if required/desired
    Image
  5. hit enter to start the test tool

If you find any bugs I suggest to report them on our public Bugtracker, as its more efficient to manage them there.
We are looking forward to your Feedback and hope the tool will support you creating mods and submitting them to ModHub.
Gruß Heizer
User avatar
w33zl
Posts: 52
Joined: Mon Apr 06, 2020 7:41 pm

Re: FS19 - TestRunner Tool for Testing Mods

Post by w33zl »

Great news! I've been thinking of developing something similar myself :) If I got a suggestion or feature request, should that be registered in the bugtracker as well?
Check out my mods: Stockman | TidyShop | Pallet Price Balancer | Buy Used Equipment

Want to report a bug or have a suggestion to any of my mods? Check out my official Farming Simulator project website. Or support me on my fan pages Facebook or Patreon .
User avatar
derSchreiner
GIANTS Software | QA Analyst
Posts: 3728
Joined: Fri Jan 01, 2010 8:02 pm
Location: Nürnberg, Franken

Re: FS19 - TestRunner Tool for Testing Mods

Post by derSchreiner »

Hi w33zl, we have planned more features, but feel free to add a feature request at the bugtracker
Grüße derSchreiner

How to report a BUG -> German / English | How to post a log file | How to upload images
Cubic Dogg
Posts: 17
Joined: Mon Mar 04, 2019 2:05 pm

Re: FS19 - TestRunner Tool for Testing Mods

Post by Cubic Dogg »

Does this work for consoles aswell or just PC
User avatar
yumi
Posts: 243
Joined: Sun Jun 19, 2016 5:48 pm
Contact:

Re: FS19 - TestRunner Tool for Testing Mods

Post by yumi »

Thanks Giants. That's great news and I hope it will help you spending more time on real quality testing when this first step is done by the modder himself.
moreRealistic fan !
FS19 "mr" on Old Farm Coutryside and Champs de France.
Give a try to ContractorMod!
Contractor Mod Beta version for FS22
All released mods source code is on Github
User avatar
SechzgerHiase
Posts: 528
Joined: Mon Aug 02, 2010 5:43 pm
Location: Bayern, Erding
Contact:

Re: FS19 - TestRunner Tool for Testing Mods

Post by SechzgerHiase »

:hi: very usefull tool, its fun to test with it :gamer:
Reallife Vlogs aus der Land- & Forstwirtschaft:

https://www.youtube.com/c/WoidHias
Mantrid
Posts: 32
Joined: Sat Sep 19, 2020 7:07 am
Contact:

Re: FS19 - TestRunner Tool for Testing Mods

Post by Mantrid »

Great stuff. A few times I've submitted a mod and it's failed with a basic error that made me wish you'd release the script/program you were using to find it. Well done :)
Website, Facebook, Youtube, ModHub, Discord user: Ford Prefect#5243 on Giants Discord
adr0102
Posts: 2
Joined: Wed May 06, 2020 9:54 pm

Re: FS19 - TestRunner Tool for Testing Mods

Post by adr0102 »

i don't know how i can install testRunner and use cmd
#MB-Trac_Power
Posts: 421
Joined: Tue Jul 03, 2018 6:21 pm

Re: FS19 - TestRunner Tool for Testing Mods

Post by #MB-Trac_Power »

Thank you!


With friendly regards
#MB-Trac_Power
User avatar
TopAce888
Posts: 17965
Joined: Tue May 15, 2018 6:29 pm
Contact:

Re: FS19 - TestRunner Tool for Testing Mods

Post by TopAce888 »

adr0102 wrote: Thu Nov 26, 2020 2:46 pm i don't know how i can install testRunner and use cmd
Just drag and drop the mod folder over the TestRunner_public.exe.
Meine LS22-Mods
Neu-Minibrunn Support Thread
___________________________________________________
log.txt richtig posten / How to post the log.txt
Giants Bilder/Image Server

Mein fraenkforfriends-Code für kostenlos 2GB zusätzlich für dich und mich: OLIW68
User avatar
cwattyeso
Posts: 1760
Joined: Tue Nov 27, 2018 3:58 pm
Location: United Kingdom
Contact:

Re: FS19 - TestRunner Tool for Testing Mods

Post by cwattyeso »

So tried to use this tool today and have run into the issue with the tool not recognising or finding the path to the game or the editor on my PC, because I have installed both the game and the editor to non-default drives and paths. I see the instructions above for doing this with cmd line or powershell but I'm still having issues with this. Is there no easier way, maybe through a separate config file that the TestTool could have users set their paths to the game.exe and editor.exe and have this saved permanently so they don't need to keep launching with cmd line or powershell and adding arguments everytime they want to use the tool?
Check out my YouTube Channel for Farming Simulator and other Gaming Let's Plays and Videos at https://www.youtube.com/c/CwattyesoGamingforFun :D

Also be sure to check out my Kick Channel for my Farming Simulator Live Streams and Multiplayer Server at https://www.kick.com/cwattyeso :gamer:

GIANTS SOFTWARE Personal Partner Code: CWATTYESO *thumbsup*
Eische
Posts: 3758
Joined: Thu Oct 18, 2018 5:17 pm

Re: FS19 - TestRunner Tool for Testing Mods

Post by Eische »

cwattyeso wrote: Thu Dec 10, 2020 1:35 pm So tried to use this tool today and have run into the issue with the tool not recognising or finding the path to the game or the editor on my PC, because I have installed both the game and the editor to non-default drives and paths. I see the instructions above for doing this with cmd line or powershell but I'm still having issues with this. Is there no easier way, maybe through a separate config file that the TestTool could have users set their paths to the game.exe and editor.exe and have this saved permanently so they don't need to keep launching with cmd line or powershell and adding arguments everytime they want to use the tool?
Did you try to put the command line into a batch (.bat) file?
You can basically put all above mentioned steps (navigating to testTool folder, ...) into the .bat file.

Later on you can simply change the path of your mod and doubleclick the .bat file. Never tested it, but should work.
Playing on PC - Win10
Ryzen 3600
RX 5500XT
16GB Ram
How to post log file
How to upload pictures
Please report bugs for FS22 using the bugtracker
Mantrid
Posts: 32
Joined: Sat Sep 19, 2020 7:07 am
Contact:

Re: FS19 - TestRunner Tool for Testing Mods

Post by Mantrid »

cwattyeso wrote: Thu Dec 10, 2020 1:35 pm I see the instructions above for doing this with cmd line or powershell but I'm still having issues with this. Is there no easier way
There is. Right-click copy the exe, then paste shortcut somewhere blank. Then right-click the new shortcut and go to properties, and add the flags in the "Target" part (on the "Shortcut" tab), after the command. It works, so you can just drop the mod folder onto the new shortcut you made :)
Website, Facebook, Youtube, ModHub, Discord user: Ford Prefect#5243 on Giants Discord
User avatar
Peppe78
Posts: 8
Joined: Mon Feb 23, 2015 9:25 am

Re: FS19 - TestRunner Tool for Testing Mods

Post by Peppe78 »

I have created a video tutorial on the testRunner to help all modders who still don't know its potential. I hope it will be useful.



Edit don_apple: fixed YouTube link.
Swilley86
Posts: 2
Joined: Sun May 16, 2021 7:34 pm

Re: FS19 - TestRunner Tool for Testing Mods

Post by Swilley86 »

this test runner tool is a joke ive gone through all the trouble shooting step by step and nothing works just keeps telling me file paths dont exist
Post Reply