This is the third of a series of savefile-related changes.
This adds early-days experimental support for a completely optional
'sfctool' utility (savefile conversion tool), to be able to export
a savefile's contents into a more portable format. There are likely
to be bugs at this stage. In this initial first-attempt, the export
format is a very simple ascii output.
NetHack can be built entirely, without also building this tool.
NetHack has no dependencies on the tool.
Attempts were made to minimize duplication of existing NetHack code.
To achieve that, unfortunately, #ifdef SFCTOOL and #ifndef SFCTOOL
had to be sprinkled around through some of the existing NetHack
source code, so that it could be re-used for building the utility.
The process for building the sfctool typically recompiles the source
files with #define SFCTOOL and a distinct object file with SF- is
produced.
sfctool notes:
Universal ctags is used and required to produce the sfctool utility.
Some targets were added to the Unix and Windows Makefiles to
facilitate the build process.
make sfctool
That should build a copy in util.
Note: At present, the Unix Makefiles do not copy sfctool over to the
NetHack playground during 'make install' or 'make update'.
Until that gets resolved by someone, The tool will
have to be manually copied there by the builder/admin if
desired.
cp util/sfctool ~/nh/install/games/lib/nethackdir/sfctool
Also, a separate Visual Studio sfctool.sln solution was written and
placed in sys/windows/vs. That has has only very limited testing.
Usage:
i) To convert an existing savefile to an exportascii format
that co-resides with the savefile:
sfctool -c savefile
That *must* be executed on the same platform / architecture /
data model that produced the save file in the first place.
ii) To unconvert an existing exportascii format export file to a
historical format savefile that can then be used by NetHack:
sfctool -u savefile
That must be executed on the same target platform / architecture /
data model that was used to build the NetHack that will
utilize the save file that results.
A Windows example:
sfctool -c Fred.NetHack-saved-game
That should result in creation of Fred.NetHack-saved-game.exportascii
from existing savefile:
%USERPROFILE%\AppData\Local\NetHack\3.7\Fred.NetHack-saved-game
A Unix example:
sfctool -c 1000wizard
That should result in creation of 1000wizard.exportascii.gz
from existing savefile in the playground save directory:
1000wizard.gz
Current Mechanics:
1. Makefile recipe, or script uses universal ctags to produce
util/sf.tags.
2. util/sftags is built and executed to read util/sf.tags and
generate: include/sfproto.h and src/sfdata.c.
3. util/sfctool is built from the following:
generated file compiled with -DSFCTOOL:
src/sfdata.c -> sfdata.o
existing files compiled with -DSFCTOOL:
util/sfctool.c -> sfctool.o
util/sfexpasc.c -> sfexpasc.o
src/alloc.c -> sf-alloc.o
src/monst.c -> sf-monst.o
src/objects.c -> sf-objects.o
src/sfbase.c -> sfbase.o
src/sfstruct.c -> sfstruct.o
src/nhlua.c -> sf-nhlua.o
util/panic.c -> panic.o
src/date.c -> sf-date.o
src/decl.c -> sf-decl.o
src/artifact.c -> sf-artifact.o
src/dungeon.c -> sf-dungeon.o
src/end.c -> sf-end.o
src/engrave.c -> sf-engrave.o
src/cfgfiles.c -> sf-cfgfiles.o
src/files.c -> sf-files.o
src/light.c -> sf-light.o
src/mdlib.c -> sf-mdlib.o
src/mkmaze.c -> sf-mkmaze.o
src/mkroom.c -> sf-mkroom.o
src/o_init.c -> sf-o_init.o
src/region.c -> sf-region.o
src/restore.c -> sf-restore.o
src/rumors.c -> sf-rumors.o
src/sys.c -> sf-sys.o
src/timeout.c -> sf-timeout.o
src/track.c -> sf-track.o
src/version.c -> sf-version.o
src/worm.c -> sf-worm.o
src/strutil.c -> strutil.o
This is the second of a series of changes related to save/restore.
No EDITLEVEL bump has been included, because although the code
is changed extensively by this, the content of the savefiles have
not been changed.
Push the use of the structlevel bwrite() and mread() function use
out of the core and into sfstruct.c. This is groundwork for upcoming
changes.
In the core, replace the bwrite() and mread() calls with the
use of type-specific savefile output (Sfo) and savefile
input (Sfi) macros. The macros are defined in a new header file
savefile.h, which also contains the prototypes for the sfo_* and
sfi_* functions that the macros ultimately expand to. The functions
themselves are in src/sfbase.c.
On C99, each Sfo or Sfi macro expansion refers directly to the
corresponding type-specific sfo_* or sfi_* function.
If C23 or later is is use, the majority (all but 3 types) of the
macros refer to a single _Generic output routine sfo(nhfp, dt, tag),
and a single _Generic input routine sfi(nhfp, dt, tag), which handles
the dispatch of the type-specific underlying functions. This was
somewhat experimental, but turned out to be practical because the
compiler would gripe if the type for a variable was not included in
the _Generic when passed as an argument, so it could be fixed.
This alters the savefile verication process by having a common set
return values for the related functions such as uptodate(),
check_version(), etc. The new return values return more information
about savefile incompatibilities, beyond failure/sucess. The
additional information will be useful for an upcoming addition.
The expanded return values are:
SF_UPTODATE (0) everything matched and looks good
SF_OUTDATED (1) savefile is outdated
SF_CRITICAL_BYTE_COUNT_MISMATCH (2) critical size count mismatch
SF_DM_IL32LLP64_ON_ILP32LL64 (3) Windows x64 savefile on x86
SF_DM_I32LP64_ON_ILP32LL64 (4) Unix 64 savefile on x86
SF_DM_ILP32LL64_ON_I32LP64 (5) x86 savefile on Unix 64
SF_DM_ILP32LL64_ON_IL32LLP64 (6) x86 savefile on Windows x64
SF_DM_I32LP64_ON_IL32LLP64 (7) Unix 64 savefile on Windows x64
SF_DM_IL32LLP64_ON_I32LP64 (8) Windows x64 savefile on Unix 64
SF_DM_MISMATCH (9) some other mismatch
The callers in the core have been adjusted to deal with the expanded
return values.
Other miscellaneous inclusions:
- go.oracle_loc -> svo.oracle_loc.
- add a bit (1UL << 30) to called SFCTOOL_BIT as groundwork
for changes to follow.
This is the first of several savefile-related changes to
follow later. This one is groundwork for those later changes.
Remove internal compression schemes (RLECOMP and ZEROCOMP)
and discard the savefile_info struct that was primarily used to
convey which internal compression schemes had been in use.
Relocate some struct definitions into appropriate header files
for use by code to come in later changes.
Remove the two struct size-related fields from version_info and
from the nmakedefs_s. Instead, include a series of bytes near the
beginning of the savefile, representing the size of each
struct or base data type that impacts the historical savefile
content. Those are referred to as the "critical bytes".
(Related note: the "you" struct required two bytes, low and high,
due to its size).
Compare those critical bytes in a savefile against the NetHack
build that is reading the savefile. This allows mismatch detection
early in the savefile-reading process, and a clean exit, rather than
proceeding to read nonsensical values from the file. Include some
feedback on what the first mismatch was when encountering
one.
For arrays stored in the savefile, use loop-logic in the core
to write/read the array elements one at a time, rather than in
a single blob. This will be required for changes to follow later.
(impacts artiexist[], artidisco[], svd.dungeons[], svl.level_info[],
svl.level.locations[][], msrooms[] field of mapseen, svb.bases[],
svb.disco[] objects[], svm.mvitals[], svs.spl_book[], svd.doors[],
go.oracle_loc[], utrack[], wgrowtime[])
This also adds data model to the long version information.
This invalidates existing save and bones files due to the changes in
the information at the start of the file.
Some variants were already using a similar approach
using a struct called 'ebones', so adopt the same naming
so NetHack-3.7, hardfought, and some variants are using
the same name.
As before there are fields in the struct that are not
currently used by NetHack-3.7, but the intent is that
hardfought save and bones files can be loaded by
NetHack-3.7 without code modification, for debugging
bug reports.
This invalidates existing save and bones files.
There are two hardfought code additions that render save and bones files incompatible
with the upstream NetHack-3.7, and that makes testing with hardfought
save and bones files more challenging than it needs to be, when
investigating and troubleshooting bug reports.
Add some unused fields to advance towards achieving save file parity with
hardfought, which is a significant source of play-testing for NetHack-3.7.
1) the elbereth field addition to u_conduct
This adds an unused placeholder field named 'hf_reserved1', at the appropriate
place in u_conduct to achieve struct field parity with the one in use on
hardfought.
2) hardfought adds a field to struct monst:
char former_rank[25]; /* for bones' ghost rank in their former life */
Instead of adding that to every monst, this adds a new mextra struct
named 'former', which currently contains the equivalent 25-character
field called 'rank' which can hold the content that was in the
former_rank[25] field. That way, the field will only be added when it
is needed.
A pull request https://github.com/k21971/NetHack37/pull/2 has been
done on hardfought to do it the same way (untested there as of yet).
Even though NetHack-3.7 does not utilize that information presently,
this will be a further step toward allowing hardfought-generated save
and bones files to be used for troubleshooting, without modification,
on a similar architecture running stock NetHack-3.7 code.
That savefile parity won't be achieved until the after the
hardfought pull-request mentioned above (or equivalent) is merged.
As this change will not be compatible with existing save and bones
files, it will be accompanied with an EDITLEVEL increment.
If the saved game menu has more than 13 games, it won't be able to
use 'n' for "new game" and 'q' for "quit". Switch to 'N' and 'Q'
instead of just using the next letters in sequence. Only resort to
next-letters if there are more than 39 games.
tty and curses handle a list of many save files via menu pagination.
X11 does so with one long page possessing a scroll bar. If there
are more than 52 entries, selection via mouse is needed beyond 'Z'.
Qt has one page without any scroll bar so won't provide access to
the full set of save files when there are too many to fit on the
screen.
Instead of a menu listing
a - hero1
b - hero2
n - New game
q - Quit
show
a - hero1-role1-race1-gend1-algn1
b - hero2-role2-race2-gend2-algn2
n - New game
q - Quit
or
a - - hero1-role1-race1-gend1-algn1
b - X hero2-role2-race2-gend2-algn2
c - D wizard-role3-race3-gend3-algn3
n - New game
q - Quit
when any game in the list wasn't saved during normal play. (Those
are sorted by character name; the playmode is just coincidence.)
The dash for 'normal' doesn't look great but -/X/D are codes used in
entries written to paniclog. The whole playmode prefix doesn't look
particularly good but I suspect that most players relying on restore
via menu won't see it.
It should work when the character name has dashes in it but that
hasn't been properly tested.
The gender and alignment suffices reflect their value at the time of
save rather than at the start of the game. That might be considered
a bug but it was easiest.
Increments EDITLEVEL; existing save and bones files are invalidated.
The g? structs had a mix of variables that were written to
the savefile, and those that were not.
For better clarity and to distinguish those that end up in
the savefile, relocate some g? variables that get written
directly to the savefile into different structs.
This updates EDITLEVEL, although technically it probably
didn't need to, since savefile contents are not changing.
Details:
gb.bases -> svb.bases
gb.bbubbles -> svb.bbubbles
gb.branches -> svb.branches
gc.context -> svc.context
gd.disco -> svd.disco
gd.dndest -> svd.dndest
gd.doors -> svd.doors
gd.doors_alloc -> svd.doors_alloc
gd.dungeon_topology -> svd.dungeon_topology
gd.dungeons -> svd.dungeons
ge.exclusion_zones -> sve.exclusion_zones
gh.hackpid -> svh.hackpid
gi.inv_pos -> svi.inv_pos
gk.killer -> svk.killer
gl.lastseentyp -> svl.lastseentyp
gl.level -> svl.level
gl.level_info -> svl.level_info
gm.mapseenchn -> svm.mapseenchn
gm.moves -> svm.moves
gm.mvitals -> svm.mvitals
gn.n_dgns -> svn.n_dgns
gn.n_regions -> svn.n_regions
gn.nroom -> svn.nroom
go.oracle_cnt -> svo.oracle_cnt
gp.pl_character -> svp.pl_character
gp.pl_fruit -> svp.pl_fruit
gp.plname -> svp.plname
gp.program_state -> svp.program_state
gq.quest_status -> svq.quest_status
gr.rooms -> svr.rooms
gs.sp_levchn -> svs.sp_levchn
gs.spl_book -> svs.spl_book
gt.timer_id -> svt.timer_id
gt.tune -> svt.tune
gu.updest -> svu.updest
gx.xmax -> svx.xmax
gx.xmin -> svx.xmin
gy.ymax -> svy.ymax
gy.ymin -> svy.ymin
Related note:
There are some pointer variables that are heads of chains that were not
moved from 'g?' to 'sv?', because they are not actually written to the
savefile directly, but the objects/monst/trap/lightsource/timer in the
chains they point to are. That can be changed, if desired.
Examples: gi.invent, gm.migrating_objs, gb.billobjs, gm.migrating_mons,
gf.ftrap, gl.light_base, gt.timer_base
Restoring was hiding unhidden mimics and if that chose an object
other than boulder of gold pieces, it called mkobj() before an
array used by that routine was initialized. The result was warning
"rnd(0) attempted" when NH_DEVEL_STATUS wasn't set to 'released' or
divide by 0 crash if it was set to that.
Restore should not be catching up for lost time when unpacking a
save file into individual level files, and if it hadn't done that
it wouldn't hide mimics who aren't currently hidden.
In addition to avoiding that, this also moves the initialization of
the offending mkobj array sooner.
[3.6 didn't use that array so wasn't susceptible to this. It is
hiding unhidden mimics during what should be a strictly bookkeeping
operation though.]
Prevent glyph change messages while loading a level file.
Otherwise a monster hiding under an item triggered a vision recalc
before light sources were linked to their sources, leading into
strange errors looking like pointer corruption.
getlev -> hide_monst -> hideunder -> newsym -> show_glyph ->
pline_xy -> vision_recalc -> do_light_sources -> get_obj_location
Add in_getlev program state variable, analogous to in_mklev
Consistent with their mythological role of punishing those who had
violated societal taboos -- oathbreakers, hosts who attacked their
guests, etc -- erinyes scale with the cumulative amount of alignment
abuse the hero has committed over the course of the game. This is
tracked separately from the alignment record, and cannot be cleared by
the hero improving her favor with her god via "good deeds" as the normal
alignment record can. Erinyes will gain abilities, levels, and attacks
as the hero's alignment abuse worsens. They will also aggravate
monsters when near the hero.
src/restore.c(275): warning: Dereferencing NULL pointer. 'otmp' contains the
same NULL value as 'first' did.
src/restore.c(402): warning: Dereferencing NULL pointer. 'mtmp' contains the
same NULL value as 'first' did.
The tracks left by hero were cleared when player saved and
restored the game, or changed levels. Now the tracks are
saved in the dungeon level, so changing levels keeps the tracks
left by hero in that level.
Also increased the length of tracks from 50 to 100, and
simplify the tracking function.
Thing not done: fade out old tracks when returning to a level.
Breaks saves and bones.
Instead of just accepting an attribute, it's now possible to
use a color, or both color and attribute, for example:
OPTIONS=menu_headings:inverse
OPTIONS=menu_headings:red
OPTIONS=menu_headings:red&underline
Default is still just inverse.
This lets the player change the menu heading color without
needing to use menu colors for them.
Also makes it so the core uses NO_COLOR instead of 0, for all
the menu lines which don't have any prefedefined color.
Tested for tty, curses, x11, qt, and win32
I realized that failed explore-mode authorization on a special-mode
saved game cannot downgrade the game mode further down to a normal game,
because this would dump the player back into a state where she has
completed some part of the game in explore mode but is eligible for the
topten list. This is even more true when the game was formerly a
wizard-mode game. Unforunately, that was the state my previous commits
left the game in.
Instead, if restoring an explore-mode or wizard-mode savegame, and the
player is authorized via sysconf for neither of those modes, fail
restoration entirely and start a new game instead. That's sort of
clunky and there could probably be more explanation provided, but it
should be an exceedingly rare occurance and I'm not sure what
alternative exists that would still honor the EXPLORERS and WIZARDS
restrictions. This shouldn't affect the way they default 'down a mode'
in other circumstances, i.e. the overwhelming majority of situations in
which EXPLORERS authorization is needed/checked.
For the same reason, I realized that the player can't be prompted
whether or not to enter explore mode, if being downgraded from a
no-longer-authorized wizmode save while explore mode is authorized. The
change from wizard mode to explore mode must be mandatory. I have also
switched that up so that it will force the change -- unfortunately, this
has the side effect of allowing the preservation of the save, but it's
more important to make sure a wizard mode game doesn't get reverted to
normal mode. They won't be able to load the save into wizard mode
anyway.
The sysconf EXPLORERS list restricting access to explore mode was being
evaluated and used when a player used the #exploremode command in-game,
or when specifying -X or OPTIONS=playmode:explore on the command line
when resuming a normal game, but not when starting an entirely new game.
When SYSCF is avilable, check for authorization early, similar to debug
mode authorization, to restrict access to explore mode to EXPLORERS
under (hopefully) all circumstances.
If there were outdated savefiles encountered during
startup, each individual one was getting a wait_synch
that required a <return> even though a message window
wasn't being used at that point.
Allow suppression of the individual per-file wait_synch()
calls on Windows, so that a single one can be done once
the selectsave processing is overwith.
This was a little messy because an indicator had to flow
down through validate(), uptodate(), etc.
There shouldn't be any change in how things behave on
any non-Windows platforms.
Adds a new lua command
des.exclusion({ type = "teleport", region = { x1,y1, x2,y2 } });
which allows defining "exclusion zones" in the level, areas where
random teleports (or falling into the level) will never place the hero.
Does not prevent targeted teleportation into the area.
Breaks saves and bones.
Reported five months ago, a save was performed while a mounted hero
was engulfed. Restore issued a warning about the engulfer being
placed on top of the steed (who shouldn't have been on the map).
The report arrived at about the same time as engulfing a riding
hero was changed to force a dismount instead of engulfing both hero
and steed so nothing further was done about it. This changes
restore to not put a steed on the map and then take it off again.
It also attempts to simplify usteed and ustuck handling during save
and restore.
Testing so far indicates that things are still working correctly.
Keep makeplural(body_part(FINGER)) crossed.
Existing save and bones files are invalidated.
Update and apply an old stashed commit: hide the casts in calls to
mread() made in restore.c. Some split lines get recombined and there
is less clutter when reading the code.
The consolidation of global variables from scattered source
files into decl.c and declared in decl.h was begun in 3.7.0.
Their placement in common files was done for centralized
initialization and potential re-initialization during a
"play again" scenario.
It wasn't really necessary for all of them to be housed in a
single huge structure to meet the "play again" requirement,
and the single huge structure has been a little unwieldy when
it comes to maintenance.
Following this commit, instead of one single extremely large structure
named 'g' to house all of the relocated global variables, they
are distributed into several ga through gz.
To make things easy for the developer, each variable is placed
into the struct corresponding to the starting letter of the variable.
That way, no lookup is required in order to know which struct houses
a particular variable, it is a simple match to the starting letter
for all the centralized global variables.
A global variable named 'amulets', would be found in ga.
ga.amulets
^ ^
A global varable named 'move', would be found in gm.
gm.moves
^ ^
A global variable named 'val_for_n_or_more' would be found in gv.
gv.val_for_n_or_more
^ ^
A global variable named 'youmonst' would be found in gy.
gy.youmonst
^ ^
Reported by paxed 8 years ago: if a bones file contains a
doppelganger imitating a unique monster, when it gets loaded
that monster ends up being marked as having been created. The
doppelganger itself will shapechange to other forms, but the
unique monster won't be created when it should be because it has
become extinct.
Report involved creating a statue of a unique monster which
yields a doppelganger in that monster's shape, then using stone
to flesh to animate the statue, dying before it changes to some
other shape, and having bones be saved.
Trap doors saved their destinations as an absolute level, rather than a
relative one, so if you loaded bones from a special level their
destinations would reflect the dungeon layout from the bones player's
game. For example, die on the Oracle level, on dlvl5, with a trap door
that goes to dlvl6. Another player gets those bones on their Oracle
level, which is dlvl8... the trap door would still go to dlvl6. Pretty
amazing trap door -- something you might see in a funhouse!
Include relative rather than absolute destinations in save and bones
files, much like stairs do, to avoid this problem.
I bumped EDITLEVEL because although this won't break save files in an
obvious way, it will interpret the (absolute) destinations in existing
save and bones files as relative, leading to some crazy long falls. :)
Stair dlevels weren't being restored with the correct values when
recovered after the game crashed, apparently because they weren't being
reset back to their 'absolute' level from a 'relative' level. I'm not
totally sure of why this affected only recovered games (maybe that's the
only time when the 'relative' stair values are used?) but this fix seems
to work.
Fixes#812
One of the drivers of this change was that screen coordinates require a
type that can hold values greater than 127. Parameters to the window
port routines require a large type in order to be able to have values
a fair bit larger than COLNO and ROWNO passed to them, particularly for
their use to the right of the map window.
This splits the uses of xchar into 3 different situations, and adjusts
their type and size:
xchar
|
-----------------------
| | |
coordxy xint16 xint8
coordxy: Actual x or y coordinates for various things (moved to 16-bits).
xint16: Same data size as coordxy, but for non-coordinate use (16-bits).
xint8: There are only a few use cases initially, where it was very
plain to see that the variable could remain as 8-bits, rather
than be bumped to 16-bits. There are probably more such cases
that could be changed after additional review.
Note: This first changed all xchar variables to coordxy. Some were
reviewed and got changed to xint16 or xint8 when it became apparent that
their usage was not for coordinates.
This increments EDITLEVEL in patchlevel.h
(user-side decisions really, but as it stands right now
user-side decisions/options are made and processed by the core)
add a parameter to add_menu so color can be passed
starting screen (Issue #783)
On 2022-06-01 12:22 p.m., NetSysFire wrote:
> Steps to reproduce:
>
>1. Get any prompt and answer it. In my case it was a horribly old
> save I forgot about or when I wiztested something and forgot
> about that save, too.
>2. See that the copyright information got overwritten by the prompt:
>
>There is already a game in progress under your name. Destroy old game? [yn] (n)
> By Stichting Mathematisch Centrum and M. Stephenson.
> Version 3.7.0-59 Unix Work-in-progress, built May 31 2022 12:28:31.
> See license for details.
>
>
> Shall I pick character's race, role, gender and alignment for you? [ynaq]
>
> Expected behavior:
>
> Redraw after a prompt was answered, so the prompt vanishes and the
> entirety of the starting screen will be shown.
>
> NetHack, Copyright 1985-2022
> By Stichting Mathematisch Centrum and M. Stephenson.
> Version 3.7.0-59 Unix Work-in-progress, built May 31 2022 12:28:31.
> See license for details.
>
>
> Shall I pick character's race, role, gender and alignment for you? [ynaq]
>
> Proposed severity: low. Not gamebreaking, it is cosmetic only and does
> not have any other consequences.
>
The Copyright notice is placed by tty internal routines writing onto
the BASE_WINDOW fairly early in the startup sequence.
The prompt to "Destroy old game? [yn] (n)" is using the in-game
routine to write to the message window at the top of the screen and
prompt there, just like in-game prompts and messages.
If the player answered 'y' to that, the prompt for
"Shall I pick character's race, role, gender and alignment..."
appeared immediately after. That one, however, is written using
the BASE_WINDOW routines in tty, like the copyright notice.
This change does the following:
It moves the copyright lines down a little bit leaving room for the
"Destroy.." prompts.
It places the "Shall I pick characters's..." prompt further down the
screen by default, leaving some room for about 3 raw_print startup
messages after the copyright notice, just in case there are any.
The "Shall I pick character's..." prompt will still appear immediately
if there is a prompt such as "Destroy old game?..."
There were a couple of other issues around raw_print startup messages
too. Those are delivered using a raw_print mechanism to ensure they
are written even if the window-port is not fully operational. However,
they were only on the screen for the blink of an eye. This call
sequence in restore.c made them disappear almost immediately:
docrt() -> cls()
Put in a mechanism to detect the presence of raw_print messages
from the early startup, and if there were some, wait for a
keypress before obliterating the unread notifications.
