Software deployment @ Euroclear SA/NV - GSE Belux

capybarabowwowSoftware and s/w Development

Oct 30, 2013 (3 years and 9 months ago)

104 views

© 2012 IBM Corporation

Session code zZS25

*****
PRELIMINARY VERSION

V *****

Software Deployment Customer Experiences

Tim Alpaerts, Euroclear


IBM System z Technical University


Berlin, Germany


May 21
-
25

2


Terminology


The old situation


The new design


The Product Catalog in TDSL


Non
-
uniform software stacks


TDSL Product Selection & Generate (build) processes


The roll
-
out (restore) process


VSAM challenges


zFS on the installation system


Agenda


3


Installation system
: one particular MVS system dedicated to installing software and
installing fixes.



End
-
User system
: all MVS systems other than the installation system.



Dead libraries
: product libraries created on the installation system by software installation,
(maybe SMP/E installed or non
-
SMP/E installed) not actively in use by an instance of the
product.



Retrofit
: after making a local correction on an end
-
user system, it needs to be incorporated
into the dead libraries for future builds.



Version
-
less dsnames
: product library dsnames that do not imply a particular version or
release of a software, they do not contain a version release qualifier.



Non
-
Uniform software stacks
: selectively activating only the needed parts of the complete
enterprise software stack on every system, mainly for the purpose of reducing the license
charge.

Terminology

4


Terminology


The old situation


The new design


The Product Catalog in TDSL


Non
-
uniform software stacks


TDSL Product Selection & Generate (build) processes


The roll
-
out (restore) process


VSAM challenges


zFS on the installation system


Agenda


5




Development systems















Production systems






















K
-
systems










each group of systems:






Test systems Paris



Active MVS target & dlibs






Inactive MVS target & dlibs











MVS SMPE,





within each group


2 target zones,


non
-
IBM


DASD was shared with CA
-
MIM


2 dlib zones



products







The old situation, overview

IBM

IBM

SMPE

Test systems Brussels

Pre
-
production Paris

Pre
-
production Brussels

7


Version
-
release qualifiers in dsnames of product libraries make system upgrades
risky.



Enqueue and reserve contention interference between lpars, instability from dead
-
locks.



No inventory nor overview of what system is using what software releases.



No obvious distinction between software product libraries and databases and work
datasets.



Locally maintained PROGxx parmlib members accumulate errors over time



Low confidence when doing IPL.


Some issues with the old set
-
up

9


Terminology


The old situation


The new design


The Product Catalog in TDSL


Non
-
uniform software stacks


TDSL Product Selection & Generate (build) processes


The roll
-
out (restore) process


VSAM challenges


zFS on the installation system


Agenda


10


Software installation needs to create dsnames (dead libraries) different from those
the driving system is using.



All software installation work concentrated on one single system dedicated for this
purpose.



All product libraries with version
-
less dsnames, to avoid widespread impact from
system upgrades.



Distinction between R/O product libraries and R/W databases immediately
apparent from the dsname.



Complete isolation of DASD between lpars in the IODF, no more cross
-
plex
sharing of dasd.

Objectives for the new design (1)

11


All software, OS and all software products are upgraded in one go.


Upfront agreement on scope and planning (IPL slots).


Integrated testing.


All the “hold actions” are performed in parallel and are repeatable.



Provide a ‘perfect’ APF
-
list and link
-
list, complete but without any needless entries.



Ensure at all times we have an SMP perfectly in
-
sync for every software rolled
-
out on every
system.



Upgrade per system if required by a freeze of the environment.



Complete inventory of software installation and deployment states.


Visibility of what is used where (licencing, cleanup of old S/W)



Installation guidelines and documentation is available the same way for all softwares.



Objectives for the new design (2)

12










K
-
systems






Production systems





















Development systems







Test systems




















each system:




&SYSR1


Active
IPL
-
set


Inactive
IPL
-
set





&SYSR2


6 model 9 vols


6 model 9 vols





&SYSR3




&OEMR1




&OEMR2




&OEMR3











End
-
user systems

The new design, overview (1)

13


IPL
-
set is restored from a (virtual) tape that was created on the installation system.



Each sysplex has two IPL
-
sets, the active set and the inactive set.



First volume of an IPL
-
set is the load device for the end
-
user system.



First volume restored from full volume dump to include the IPL text in the dump, the other
volumes are dumped logically by dataset.



Second volume contains an ICF user catalog for the VSAM datasets on the IPL
-
set.



IPL
-
set is protected against updates by DFP OPEN exit IFG0EX0B, datasets can only be
opened for INPUT.



The new design, the IPL
-
set

15


Datasets on the IPL
-
set are cataloged with symbolic volsers defined in IEASYMxx parmlib
member with substrings from &SYSR1:


SYSDEF


SYMDEF(&SYSR2.='&SYSR1(1:5)2')


SYMDEF(&SYSR3.='&SYSR1(1:5)3')


SYMDEF(&OEMR1.='&SYSR1(1:5)5')


SYMDEF(&OEMR2.='&SYSR1(1:5)6')


SYMDEF(&OEMR3.='&SYSR1(1:5)7')


When MVS IPL processing sets &SYSR1, the volser of the IPL device to "IPCAA1",

then IEASYMxx sets



&SYSR2. = "IPCAA2"


&SYSR3. = "IPCAA3"


&OEMR1. = "IPCAA5"


&OEMR2. = "IPCAA6"


&OEMR3. = "IPCAA7"


The new design, the IPL
-
set, continued

16










K
-
systems






Production systems





















Development systems







Test systems










Level
-
set
,

6 model 9 volumes








each system:




&SYSR1


Active
IPL
-
set


Inactive
IPL
-
set





&SYSR2


6 model 9 vols


6 model 9 vols





&SYSR3




&OEMR1




&OEMR2




&OEMR3





Installation system





End
-
user systems

The new design, overview (2)

17


An exact image of the software for the end
-
user systems.



Generated by the TDSL application on the software installation system.



A ‘contents’ dataset is added with the list of all products and libraries on the LS.



The non
-
VSAM datasets on the LS are not cataloged, VSAM datasets get a
special temporary HLQ.



Dsnames on the LS have no version
-
release qualifier (limited exceptions are
allowed),



Exists only on the software installation system.



The new design, the Level
-
set

19









K
-
systems


REF
-
set
, SMS storage pool


Production systems












REFx datasets









Development systems







Test systems





TDSL





Level
-
set
,

6 model 9 volumes








each system:




&SYSR1


Active
IPL
-
set


Inactive
IPL
-
set





&SYSR2


6 model 9 vols


6 model 9 vols





&SYSR3




&OEMR1




&OEMR2




&OEMR3





Installation system



End
-
user systems

The new design, overview (3)

20


A large SMS
-
pool with dead
-
libraries



All the REF
-
datasets created by software installation go in this pool



ACS routines direct all allocations of REFx datasets to this pool



All REF
-
dsnames contain a version
-
release qualifier



Mostly vendor product libraries, but also Euroclear ‘config’ libraries



Exists only on the central software installation system


The new design, the REF
-
set

21









K
-
systems


REF
-
set
, SMS storage pool


Production systems












REFx datasets









Development systems







Test systems





TDSL





Level
-
set
,

6 model 9 volumes








each system:




&SYSR1


Active
IPL
-
set


Inactive
IPL
-
set





&SYSR2


6 model 9 vols


6 model 9 vols





&SYSR3




&OEMR1




&OEMR2




&OEMR3





Installation system



End
-
user systems

The new design, RENAME

22


Dsnames in the REF
-
set need to be renamed when they are copied to the Level
Set.



Standard processing: rename removes the version release qualifier, i.e.




REFN.T119A.OPS.LOAD








REFN / SYSN dataset:

SYSN.OPS.LOAD





one qualifier dropped







REF1.T113A.ZOS.LINKLIB








REF1 / SYS1 dataset:

SYS1.LINKLIB





two qualifiers dropped






The new design, rename rules (1)

23



The standard rename does not support two or more releases side by side.





Keep
-
Qualifier processing: rename keeps the version release qualifier, i.e.


REFN.T119A.OPS.LOAD









SYSN.T119A.OPS.LOAD





Keep
-
Qualifier processing is
never done

for SYS1 datasets:



REF1.T113A.ZOS.LINKLIB



SYS1.LINKLIB



The new design, rename rules (2)

24


Terminology


The old situation


The new design


The Product Catalog in TDSL


Non
-
uniform software stacks


TDSL Product Selection & Generate (build) processes


The roll
-
out (restore) process


VSAM challenges


zFS on the installation system


Agenda


25



TDSL application handles software deployment on the installation system.



TDSL Product Catalog describes all products and the datasets belonging to each
product in SQL tables.



Each product (and version plus maintenance level) is identified by an 8
-
char
acronym.



Product Catalog has attributes at the level of the product and at the level of the
dataset.



Products can be duplicated in the Product Catalog to apply maintenance

The TDSL Product Catalog

26


The TDSL Product Catalog, screenshot

27


The TDSL Product Catalog, screenshot

Each product is identified by
an 8
-
char acronym

Filter line provides a fast
navigation facility

More attributes on the right
not in this screenshot

LR indicator

28


The TDSL Dataset List screenshot

29



Dataset Type
:


INSTALL

product installation jcl

SMP


the CSI clusters, SMPPTS, SMPMTS, SMPLOG, etc.

DLIB


SMP dlibs, 100% inert for TDSL

TARGET

SMP/E target library or other distributable dataset

VTARGET

VSAM target dataset

UTARGET

z/fs linear, always keeps version
-
release qualifier for automount

NODIST

not distributed, but it contributes to the PROGxx member




APF


dataset should be APF on end
-
user system



LNK


dataset should be link
-
listed on end
-
user system
.


The TDSL Product Catalog, dataset attributes

31

The TDSL Product Catalog, dataset attributes, continued


DSTVOL:


a symbolic volser like &SYSR1, &SYSR2 ...






In the dataset list of the z/OS 1.13 A product:




&SYSR1







REF1.T113A.ZOS.NUCLEUS


&SYSR1



REF1.T113A.ZOS.LINKLIB


&SYSR1



&SYSR2



REF1.T113A.ZOS.IMW.SIMWSDCK &SYSR2



REF1.T113A.ZOS.OMEG.TKANMOD &SYSR3













&SYSR3



Why fix the location of target datasets on the Level
-
set?


-

to ensure SYS1.NUCLEUS etc. on &SYSR1 volume


&OEMR1

-

fewer catalog updates on the end
-
user system





32


The TDSL product catalog, screenshot of z/OS 1.13 dataset list

33


Terminology


The old situation


The new design


The Product Catalog in TDSL


Non
-
uniform software stacks


TDSL Product Selection & Generate (build) processes


The roll
-
out (restore) process


VSAM challenges


zFS on the installation system


Agenda


34









swap validation







production

systems











test

systems












development



production K
-
systems

systems











Systems using the same software stack:





EB systems

EF systems





SSE systems

K systems


Installation system

Non
-
uniform software stacks (1)

35


Why not deploy all software on all systems?


Procurement dept
. Licensing all software on all systems needlessly expensive.


Audit dept.

libraries in APF list for software products that will never be used.



Why not deploy software per
-
system?


Hard to maintain the multitude of parmlib members (i.e. PROGxx members)

Many different combinations to validate before a system upgrade.



A solution in
-
between is needed.


All software will be present on all systems, but we make the libraries APF &
linklisted only on those systems where it is licensed.


To do this we add a
where
-
used attribute
to each product in the product catalog.

Non
-
uniform software stacks (2)

36

Non
-
uniform software stacks (3)

7 flags attributed to each product, each flag is on (used in this region) or off (not used in region)















SSE

PROGEG

EF

PROGEF



the libraries of product

CORT660A
will be included in


EP

PROGEP



PROGEG, PROGEF, PROGEP, PROGEN
and

PROGEB

ENL

PROGEN



but not in

PROGEK
or

PROGES

K

PROGEK

BSOF

PROGEB

SY90

PROGES

37


Why create PROG members with a program?



Software installers know what datasets need to be APF or Link
-
listed,

it’s in the product installation instructions
-
> can be entered in TDSL.




We keep lists of datasets we don’t distribute, but that need to be in the PROG
member.




TDSL has al the info it needs to create APF lists and Link lists for the End
-
User
systems.


Creating PROG members

38

Creating PROG members (2)


39

Creating PROG members (3)


40


Terminology


The old situation


The new design


The Product Catalog in TDSL


Non
-
uniform software stacks


TDSL Product Selection & Generate (build) processes


The roll
-
out (restore) process


VSAM challenges


zFS on the installation system


Agenda


41


Choose software products to go onto a Level
-
Set.



An SQL query finds all unique DSTVOLs



Engineer specifies the ‘real’ volsers and device numbers for each symbolic
DSTVOLs,

i.e. &SYSR1 will be build on real volser L13A11



The dialog produces jcl to build the LS with the selected software products.

Only the TARGET datasets will be copied to the

LS


TDSL Generate steps

43


The TDSL Generate, screenshot

44


The problem:



ALL REF
-
Set libraries have secondary space so SMP APPLY won’t fail with
abend x37.



On the Level
-
Set, Link list libraries should have zero secondary allocation, to
prevent an extent being added when someone adds/updates a member.




The solution:



DFDSS dataset copy will copy every library into a single extent.


Copy with ALLDATA & ALLEXCP will preserve the allocated space.


A homegrown utility clears the secondary allocation value in the VTOC of the
Level
-
Set.


The treatment of link list datasets, space allocation

45


Terminology


The old situation


The new design


The Product Catalog in TDSL


Non
-
uniform software stacks


TDSL Product Selection & Generate (build) processes


The roll
-
out (restore) process


VSAM challenges


zFS on the installation system


Agenda


46


Each sysplex has an active and inactive IPL
-
set with associated master catalogs:



ACTIVE


ACTIVE


INACTIVE



INACTIVE

MASTER CATALOG

IPL
-
SET


IPL
-
SET



MASTER CATALOG





IPCAA1



IPCAB1








CATALOG.PLXP.IPCAA1.MASTER



CATALOG.PLXP.IPCAB1.MASTER



Inactive IPL
-
set is formatted, ICKDSF INIT


New master catalog is delete/defined


A dump is restored onto the inactive IPL
-
set


REPRO NOMREGECAT of the active master catalog to the new one


Inactive master catalog is fixed for new
-

or changed datasets


IPL from the inactive IPL
-
set.


Roll
-
out on the end
-
user system

47


This slide just for the notes

48


Terminology


The old situation


The new design


The Product Catalog in TDSL


Non
-
uniform software stacks


TDSL Product Selection & Generate (build) processes


The roll
-
out (restore) process


VSAM challenges


zFS on the installation system


Agenda


49


VSAM datasets need to be cataloged for all processing.

Interferece with the driving system when copying VSAM datasets w same name
.



All IDCAMS processing requires a correct VSAM ‘back
-
pointer’







Catalog volume




Volume with VSAM dataset






forward pointer



ICF catalog





VVDS



SYSV.OMVS.ROOT.ZFS





VVR





back pointer







VSAM dataset SYSV.OMVS.ROOT.ZFS

The Challenges of VSAM datasets

50

On the
installation system
:


Define a new ICF usercatalog on one of the LS volumes


Define an alias for a new temporary HLQ with RELATE to the new usercat, i.e. SYSDEF


Copy the VSAM datasets to the LS with RENAMEUNCONDITIONAL, REFx


SYSDEF


Level
-
set is dumped to tape.





On the
end
-
user system
:


Define a new ICF usercatalog on one of the IPLset volumes


Delete/define the SYSDEF alias


Restore VSAM clusters with the SYSDEF HLQ from tape (no rename)


IDCAMS ALTER all SYSDEF dsnames in the user catalog to the normal HLQ for VSAM




The distribution of VSAM datasets (1)

51


An alias with symbolic relate accomplishes the same ‘volume switching function’
as the symbolic volumes in the catalog do for non
-
vsam datasets:



DEFINE USERCATALOG



(NAME(CATALOG.SYSV.IPCAA1)




IPL
-
set A


ICFCATALOG



VOLUME(IPCAA2)



CYLINDERS(1 1)



DEFINE USERCATALOG
-



(NAME(CATALOG.SYSV.IPCAB1)




IPL
-
set B


ICFCATALOG



VOLUME(IPCAB2)



CYLINDERS(1 1)


DEFINE ALIAS(NAME(SYSV)



SYMBOLICRELATE(CATALOG.SYSV.&SYSR1))



The distribution of VSAM datasets (2)

52



LISTCAT ENTRY(SYSV) ALIAS ALL


ALIAS
---------

SYSV


IN
-
CAT
---

CATALOG.PLXP.IPCAA1.MASTER


HISTORY


RELEASE
----------------
2


ASSOCIATIONS


SYMBOLIC
-
CATALOG.SYSV.&SYSR1


RESOLVED
-
CATALOG.SYSV.IPCAA1


The distribution of VSAM datasets (3)

53


Terminology


The old situation


The new design


The Product Catalog in TDSL


Non
-
uniform software stacks


TDSL Product Selection & Generate (build) processes


The roll
-
out (restore) process


VSAM challenges


zFS on the installation system


Agenda


54

Risks of APPLY on statically mounted file systems:



If the wrong file system is mounted, an APPLY job could update USS components
belonging to a different SMP/E zone than the OS libraries.



Potentially there could be data not hardened in the file system when a R/W
mounted file system is copied with dfDSS.


How automounted file systems can help:



USS path in dddef’s can contain a zFS qualifier that makes automount mount the
file system corresponding to that zone.



Automount will unmount file systems that have not been accessed for some
amount of time.

This reduces the time window during which uncommitted data can exist.

z/FS file systems on the installation system (1)

56


/etc/auto.master:


/SERVICE/DB2/DB2BASE /etc/service_db2_db2base.map



/etc/service_db2_db2base.map


name *

type ZFS

filesystem REFN.<uc_name>.DB2.DB2BASE.ZFS

mode rdwr

duration 60

delay 10

security Yes

setuid no





DDDEF SDSNABIN in target zone TD1010A contains USS path:


'/SERVICE/DB2/DB2BASE/
T10101A
/bin/IBM/'

z/FS file systems on the installation system (2)

57

The SMP/E APPLY job:

//SMPCNTL DD *





SET BDY(TD1010A) .





APPLY SELECT(…) .


When SMP/E opens a file in the SDSNABIN directory it will cause the automount
daemon to mount


REFN.T1010A.DB2.DB2BASE.ZFS



onto mount point


/SERVICE/DB2/DB2BASE/T10101A



Automount replaces <uc_name> in the map file with the path
-
qualifier from the
SMP/E dddef path to obtain the zFS dsname:



i.e.

Filesystem

REFN.<uc_name>.DB2.DB2BASE.ZFS



becomes

REFN.
T1010A
.DB2.DB2BASE.ZFS




z/FS file systems on the installation system (3)

58

z/FS file systems on the installation system (4)


A Loose End


How to distribute large zFS file
-
systems that need SMS extended format data
class attribute?

An SMS managed volume in the IPL
-
set is most unpractical.



59

The End




Question Time.




64

Reporting on changes in the field




At levelset generation time a hash is created for all datasets.


Reporting is available to identify all changes between the levelset and a resident set
deployed in a user system.