This is the third of a series of savefile-related changes.
This adds early-days experimental support for a completely optional
'sfctool' utility (savefile conversion tool), to be able to export
a savefile's contents into a more portable format. There are likely
to be bugs at this stage. In this initial first-attempt, the export
format is a very simple ascii output.
NetHack can be built entirely, without also building this tool.
NetHack has no dependencies on the tool.
Attempts were made to minimize duplication of existing NetHack code.
To achieve that, unfortunately, #ifdef SFCTOOL and #ifndef SFCTOOL
had to be sprinkled around through some of the existing NetHack
source code, so that it could be re-used for building the utility.
The process for building the sfctool typically recompiles the source
files with #define SFCTOOL and a distinct object file with SF- is
produced.
sfctool notes:
Universal ctags is used and required to produce the sfctool utility.
Some targets were added to the Unix and Windows Makefiles to
facilitate the build process.
make sfctool
That should build a copy in util.
Note: At present, the Unix Makefiles do not copy sfctool over to the
NetHack playground during 'make install' or 'make update'.
Until that gets resolved by someone, The tool will
have to be manually copied there by the builder/admin if
desired.
cp util/sfctool ~/nh/install/games/lib/nethackdir/sfctool
Also, a separate Visual Studio sfctool.sln solution was written and
placed in sys/windows/vs. That has has only very limited testing.
Usage:
i) To convert an existing savefile to an exportascii format
that co-resides with the savefile:
sfctool -c savefile
That *must* be executed on the same platform / architecture /
data model that produced the save file in the first place.
ii) To unconvert an existing exportascii format export file to a
historical format savefile that can then be used by NetHack:
sfctool -u savefile
That must be executed on the same target platform / architecture /
data model that was used to build the NetHack that will
utilize the save file that results.
A Windows example:
sfctool -c Fred.NetHack-saved-game
That should result in creation of Fred.NetHack-saved-game.exportascii
from existing savefile:
%USERPROFILE%\AppData\Local\NetHack\3.7\Fred.NetHack-saved-game
A Unix example:
sfctool -c 1000wizard
That should result in creation of 1000wizard.exportascii.gz
from existing savefile in the playground save directory:
1000wizard.gz
Current Mechanics:
1. Makefile recipe, or script uses universal ctags to produce
util/sf.tags.
2. util/sftags is built and executed to read util/sf.tags and
generate: include/sfproto.h and src/sfdata.c.
3. util/sfctool is built from the following:
generated file compiled with -DSFCTOOL:
src/sfdata.c -> sfdata.o
existing files compiled with -DSFCTOOL:
util/sfctool.c -> sfctool.o
util/sfexpasc.c -> sfexpasc.o
src/alloc.c -> sf-alloc.o
src/monst.c -> sf-monst.o
src/objects.c -> sf-objects.o
src/sfbase.c -> sfbase.o
src/sfstruct.c -> sfstruct.o
src/nhlua.c -> sf-nhlua.o
util/panic.c -> panic.o
src/date.c -> sf-date.o
src/decl.c -> sf-decl.o
src/artifact.c -> sf-artifact.o
src/dungeon.c -> sf-dungeon.o
src/end.c -> sf-end.o
src/engrave.c -> sf-engrave.o
src/cfgfiles.c -> sf-cfgfiles.o
src/files.c -> sf-files.o
src/light.c -> sf-light.o
src/mdlib.c -> sf-mdlib.o
src/mkmaze.c -> sf-mkmaze.o
src/mkroom.c -> sf-mkroom.o
src/o_init.c -> sf-o_init.o
src/region.c -> sf-region.o
src/restore.c -> sf-restore.o
src/rumors.c -> sf-rumors.o
src/sys.c -> sf-sys.o
src/timeout.c -> sf-timeout.o
src/track.c -> sf-track.o
src/version.c -> sf-version.o
src/worm.c -> sf-worm.o
src/strutil.c -> strutil.o
This is the second of a series of changes related to save/restore.
No EDITLEVEL bump has been included, because although the code
is changed extensively by this, the content of the savefiles have
not been changed.
Push the use of the structlevel bwrite() and mread() function use
out of the core and into sfstruct.c. This is groundwork for upcoming
changes.
In the core, replace the bwrite() and mread() calls with the
use of type-specific savefile output (Sfo) and savefile
input (Sfi) macros. The macros are defined in a new header file
savefile.h, which also contains the prototypes for the sfo_* and
sfi_* functions that the macros ultimately expand to. The functions
themselves are in src/sfbase.c.
On C99, each Sfo or Sfi macro expansion refers directly to the
corresponding type-specific sfo_* or sfi_* function.
If C23 or later is is use, the majority (all but 3 types) of the
macros refer to a single _Generic output routine sfo(nhfp, dt, tag),
and a single _Generic input routine sfi(nhfp, dt, tag), which handles
the dispatch of the type-specific underlying functions. This was
somewhat experimental, but turned out to be practical because the
compiler would gripe if the type for a variable was not included in
the _Generic when passed as an argument, so it could be fixed.
This alters the savefile verication process by having a common set
return values for the related functions such as uptodate(),
check_version(), etc. The new return values return more information
about savefile incompatibilities, beyond failure/sucess. The
additional information will be useful for an upcoming addition.
The expanded return values are:
SF_UPTODATE (0) everything matched and looks good
SF_OUTDATED (1) savefile is outdated
SF_CRITICAL_BYTE_COUNT_MISMATCH (2) critical size count mismatch
SF_DM_IL32LLP64_ON_ILP32LL64 (3) Windows x64 savefile on x86
SF_DM_I32LP64_ON_ILP32LL64 (4) Unix 64 savefile on x86
SF_DM_ILP32LL64_ON_I32LP64 (5) x86 savefile on Unix 64
SF_DM_ILP32LL64_ON_IL32LLP64 (6) x86 savefile on Windows x64
SF_DM_I32LP64_ON_IL32LLP64 (7) Unix 64 savefile on Windows x64
SF_DM_IL32LLP64_ON_I32LP64 (8) Windows x64 savefile on Unix 64
SF_DM_MISMATCH (9) some other mismatch
The callers in the core have been adjusted to deal with the expanded
return values.
Other miscellaneous inclusions:
- go.oracle_loc -> svo.oracle_loc.
- add a bit (1UL << 30) to called SFCTOOL_BIT as groundwork
for changes to follow.
+15 wasn't very impactful in the late game and late mid-game, but
was much too lethal in the early game (wearing the ring for a while
near the start of the game would make the game unwinnable as very
out-of-depth monsters spawned, and they would still be there even
after removing the ring and usually capable of one-shotting an
early-game character). This commit changes it to a doubling of
level difficulty rather than a flat increase: that makes it more
relevant in the late-game where a +25 or +50 might potentially have
an impact, and more survivable in the early game (although it still
spawns monsters that are difficult for the point in the game, there
is now a chance that you might survive long enough to be able to
take the ring off and clear off all the out-of-depth monsters that
spawned).
'heaputil' is producing a lot of complaints. This fixes one of them,
about freeing memory that was never allocated. In this case, it's
when removing an overview annotation for a level. The annotation
is using dupstr_n() and not being recorded due to dupstr_n() being
placed after MONITOR_HEAP undefines the macro that overrides alloc().
There's only one use of dupstr_n(), and its length checking isn't
needed there, so just switch to dupstr() and comment out the
implementation of dupstr_n(). I left the prototype in extern.h;
that's harmless.
If dupstr_n() needs to be resurrected, a second MONITOR_HEAP-aware
version should be implemented, with corresponding macro to choose
which one to use.
This is the first of several savefile-related changes to
follow later. This one is groundwork for those later changes.
Remove internal compression schemes (RLECOMP and ZEROCOMP)
and discard the savefile_info struct that was primarily used to
convey which internal compression schemes had been in use.
Relocate some struct definitions into appropriate header files
for use by code to come in later changes.
Remove the two struct size-related fields from version_info and
from the nmakedefs_s. Instead, include a series of bytes near the
beginning of the savefile, representing the size of each
struct or base data type that impacts the historical savefile
content. Those are referred to as the "critical bytes".
(Related note: the "you" struct required two bytes, low and high,
due to its size).
Compare those critical bytes in a savefile against the NetHack
build that is reading the savefile. This allows mismatch detection
early in the savefile-reading process, and a clean exit, rather than
proceeding to read nonsensical values from the file. Include some
feedback on what the first mismatch was when encountering
one.
For arrays stored in the savefile, use loop-logic in the core
to write/read the array elements one at a time, rather than in
a single blob. This will be required for changes to follow later.
(impacts artiexist[], artidisco[], svd.dungeons[], svl.level_info[],
svl.level.locations[][], msrooms[] field of mapseen, svb.bases[],
svb.disco[] objects[], svm.mvitals[], svs.spl_book[], svd.doors[],
go.oracle_loc[], utrack[], wgrowtime[])
This also adds data model to the long version information.
This invalidates existing save and bones files due to the changes in
the information at the start of the file.
I'm not really sure about this one. insert_branch(branch,) is
specified as not accepting a Null pointer and doesn't have any
defense against it, but the know level setup seems to allow a null
pointer through. I'm not sure whether this is the right fix.
gcc has recognized various "magic comments" for white-listing
occurrences of implicit fallthrough in switch statements for
a long time:
The range and shape of "falls through" comments accepted are
contingent upon the level of the warning. (The default level is =3.)
-Wimplicit-fallthrough=0 disables the warning altogether.
-Wimplicit-fallthrough=1 treats any kind of comment as a "falls through" comment.
-Wimplicit-fallthrough=2 essentially accepts any comment that contains something
that matches (case insensitively) "falls?[ \t-]*thr(ough|u)" regular expression.
-Wimplicit-fallthrough=3 case sensitively matches a wide range of regular
expressions, listed in the GCC manual. E.g., all of these are accepted:
/* Falls through. */
/* fall-thru */
/* Else falls through. */
/* FALLTHRU */
/* ... falls through ... */
etc.
-Wimplicit-fallthrough=4 also, case sensitively matches a range of regular
expressions but is much more strict than level =3.
-Wimplicit-fallthrough=5 doesn't recognize any comments.
Plenty of other compilers did not recognize the gcc comment convention,
and up until now the compiler warning for detecting unintended
fallthrough had to be suppressed on other compilers. That's because the code
in NetHack has been relying on the gcc approach, and only the gcc approach.
The C23 standard introduces an attribute [[fallthrough]] for the
functionality, when implicit fallthrough warnings have been enabled.
Several popular compilers already support that, or a very similar attribute
style approach, today, even ahead of their C23 support:
C compiler whitelist approach
--------------------------- -------------------------------------
C23 conforming compilers [[fallthrough]]
clang versions supporting
standards prior to
C23 __attribute__((__fallthrough__))
Microsoft Visual Studio
since VS 2022 17.4.
The warning C5262 controls
whether the implict
fallthrough is detected and
warned about with
/std:clatest. [[fallthrough]]
This adds support to NetHack for the attribute approach by inserting a
macro FALLTHROUGH to the existing cases that require white-listing, so
other compilers can analyze things too.
The definition of the FALLTHROUGH macro is controlled in include/tradstdc.h.
The gcc comment approach has also been left in place at this time.
The g? structs had a mix of variables that were written to
the savefile, and those that were not.
For better clarity and to distinguish those that end up in
the savefile, relocate some g? variables that get written
directly to the savefile into different structs.
This updates EDITLEVEL, although technically it probably
didn't need to, since savefile contents are not changing.
Details:
gb.bases -> svb.bases
gb.bbubbles -> svb.bbubbles
gb.branches -> svb.branches
gc.context -> svc.context
gd.disco -> svd.disco
gd.dndest -> svd.dndest
gd.doors -> svd.doors
gd.doors_alloc -> svd.doors_alloc
gd.dungeon_topology -> svd.dungeon_topology
gd.dungeons -> svd.dungeons
ge.exclusion_zones -> sve.exclusion_zones
gh.hackpid -> svh.hackpid
gi.inv_pos -> svi.inv_pos
gk.killer -> svk.killer
gl.lastseentyp -> svl.lastseentyp
gl.level -> svl.level
gl.level_info -> svl.level_info
gm.mapseenchn -> svm.mapseenchn
gm.moves -> svm.moves
gm.mvitals -> svm.mvitals
gn.n_dgns -> svn.n_dgns
gn.n_regions -> svn.n_regions
gn.nroom -> svn.nroom
go.oracle_cnt -> svo.oracle_cnt
gp.pl_character -> svp.pl_character
gp.pl_fruit -> svp.pl_fruit
gp.plname -> svp.plname
gp.program_state -> svp.program_state
gq.quest_status -> svq.quest_status
gr.rooms -> svr.rooms
gs.sp_levchn -> svs.sp_levchn
gs.spl_book -> svs.spl_book
gt.timer_id -> svt.timer_id
gt.tune -> svt.tune
gu.updest -> svu.updest
gx.xmax -> svx.xmax
gx.xmin -> svx.xmin
gy.ymax -> svy.ymax
gy.ymin -> svy.ymin
Related note:
There are some pointer variables that are heads of chains that were not
moved from 'g?' to 'sv?', because they are not actually written to the
savefile directly, but the objects/monst/trap/lightsource/timer in the
chains they point to are. That can be changed, if desired.
Examples: gi.invent, gm.migrating_objs, gb.billobjs, gm.migrating_mons,
gf.ftrap, gl.light_base, gt.timer_base
Add the shop type variations used for automatically generated
annotations to the shop structure and get rid of the switch that
has been being used to pick them.
new .h files: hacklib.h selvar.h stairs.h
new .c files: calendar.c, getpos.c, report.c, selvar.c, stairs.c,
strutil.c, wizcmds.c
cleanup of hacklib.c and mdlib.c
hacklib contains functions that do not have to link with the core
relocate wiz commands from cmd.c to wizcmds.c
relocate CRASHREPORT stuff to report.c
relocate getpos stuff from do_name.c to getpos.c
remove temporary struct definition from extern.h
cross-compile PRE-section split into cross-pre1.370 and cross-pre2.370
Windows sys/windows/Makefile.nmake and sys/windows/Makefile.mingw32 and
visual studio project file updates
Unix sys/unix/Makefile.src, sys/unix/Makefile.utl
populate selvar.c and selvar.h
build on MS-DOS (not cross-compile) Makefile updates
for sys/msdos/Makefile.GCC (untested)
vms updates for above (untested)
- add nhl_pcall_handle() to wrap all nhl_pcall calls that didn't check
return value and either panic() or impossible()
- add --loglua (unix only) to dump Lua memory and steps info to livelog
- remove old logging
- set memory and step limits on all Lua VMs
Update several places where lazy lastseentyp[] might be an issue.
I think it isn't updated in a timely fashion when newsym() shows
a spot covered by an object or trap, but didn't manage to find any
cases where that caused a problem. This is more in the nature of
a precaution.
Callgrind showed recalc_mapseen was three times more expensive (in terms
of instructions read) than anything else in our codebase. It was being
called in every vision change, re-evaluating the last seen map terrain
type for every map location in sight.
Remove updating the lastseen info in the vision code, and make a small
change so newsym() uses update_lastseentyp.
From my short tests, this seems to work correctly ...
u_on_newpos() bit: player can't see the map while swallowed so hero
can't see objects on the map, hence shouldn't gain more info about
any generic objects if engulfer moves closer to some.
At the moment engulfer movement is manipulating <u.ux,u.uy> directly
rather than going through u_on_newpos(), but that's about to change.
Otherwise a clipped map doesn't get updated properly until the hero
is eventually expelled.
When using the 'm' prefix with #overview to get a menu of visited
levels and then picking one to annotate, replace the generic prompt
"what do you want to call this dungeon level?" with more specific
location information. Location details are visible while within the
menu but as soon as you choose something that goes away.
Sometimes I annotate a level with a note like "watch out, chameleon
below", which is useful to remind myself of some danger or thing to
remember when returning to the level -- but if saving and restoring on
the level itself there's no reminder of that annotation. If you restore
on a level with an annotation, print it as part of the "welcome back"
message.
Simplify suppression of highlighting for menu header lines during end
of game disclosure. Didn't actually affect as many things as I was
expecting.
Plus a bit left out of the optfn_dogname() parsing commit.
Instead of just accepting an attribute, it's now possible to
use a color, or both color and attribute, for example:
OPTIONS=menu_headings:inverse
OPTIONS=menu_headings:red
OPTIONS=menu_headings:red&underline
Default is still just inverse.
This lets the player change the menu heading color without
needing to use menu colors for them.
Also makes it so the core uses NO_COLOR instead of 0, for all
the menu lines which don't have any prefedefined color.
Tested for tty, curses, x11, qt, and win32
Reported by entrez, some putstr() to text window got changed to
add_menu_str(). I didn't test with curses; with tty some headers
ended up in limbo: "Artifacts" header for '` a y' (wizard mode show
artifacts, something I had forgotten even existed) and also monster
class headers for 'm #vanquished by-class' (available to everyone).
Qt lost them too, but at least it didn't panic.
Not due to over-simplification: end of game disclosure suppresses
header line highlighting, except when disclosing final inventory.
Change it to do so, although it would be simpler overall to just not
bother with any menu_headings highlight suppression.
Reported by entrez: it was possible for #overview to show a line of
just "." if a temple was known and its altar was unknown and no other
features such as thrones or fountains were known on the level.
It now lists "M temples and N altars" when both are present and the
case that yielded "." becomes "a temple". That's an improvement but
there might be edge cases it gets wrong. A listing of "a temple and
an altar" is ambiguous because there isn't any way to tell whether the
altar it mentions is inside the temple. That seems acceptable to me.
I think it should include more alignment information about temples and
altars, instead of just adding "to <your god>" when all known altars
are of hero's alignment, but this doesn't attempt to address that.
Adds a new lua command
des.exclusion({ type = "teleport", region = { x1,y1, x2,y2 } });
which allows defining "exclusion zones" in the level, areas where
random teleports (or falling into the level) will never place the hero.
Does not prevent targeted teleportation into the area.
Breaks saves and bones.
When returning to play from within the tutorial, remove the level files
similar to how they're discarded for the rest of the dungeon when going
into the endgame. It turned out to be a bit messier than anticipated.
The dungeon.c bit is sufficient for #overview, which now hides regular
level 1 while in the tutorial and hides all tutorial levels once exited.
Those will still appear in end-of-game disclosure.
The FIXME comment noted that builds_up would return an incorrect false
value for a dungeon branch that builds upwards but is only 1 level, but
that this is a latent problem because no such branch exists in NetHack.
Such a branch does exist in xNetHack, and it causes the debug fuzzer to
crash ("mon_arrive: no corresponding portal" because it can't find the
correct-direction stairs), so I figured I might as well fix it upstream.
