You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 18 Next »

In order to perform this migration, you need to request Openstack Application Credentials EWC - How to request Openstack Application Credentials


User procedure

  1. backup existing ldap server before starting (using Openstack credentials) ADD LINK
  2. add port 636 TCP.749 TCP,464 UDP to ldap security group
  3. Run workflow run workflow to create dns reverse zone into LDAP machine to create the reverse hosted zone (if missing)

    ipa dnszone-add --name-from-ip=10.0.0.63 or using IP range (https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/managing-reverse-dns-zones)

  4. create LDAP replica instance type to move from centos7 to rocky 8 (see Migrate from Centos7 to Rocky 8 section below)
  5. run workflow to switch IP interfaces between LDAP
  6. Check everything is fine (deploy a new machine, login to other machines using DNS)
  7. Remove old LDAP
  8. Repeat from step 2. to move from rocky 8 to rocky 9 (and future)


Migrate from Centos7 to Rocky 8

rocky 8

Creating a new LDAP to migrate to

  1. deploy a new machine with
    1. rocky 8
    2. new ldap security group
    3. plan eo1.medium
  2. become sudo
  3. stop and disable firewalld on the machine systemctl stop firewalld && systemctl disable firewalld
  4. add old ldap IP and domain of the machine acting as LDAP in one line to /etc/hosts on this machine
  5. [test234@test-podman2 ~]$ cat /etc/hosts
<!-- BEGIN ANSIBLE MANAGED BLOCK -->
10.0.0.133 test-podman2.eumetsat.sandbox.ewc
10.0.0.63 <OLD LDAP complete domain (e.g. ldap.eumetsat.sandbox.ewc)>
<!-- END ANSIBLE MANAGED BLOCK -->
  6. Install dependencies
    1. for rocky 8:

      1. sudo yum -y update

      2. sudo yum module reset idm:DL1

      3. sudo yum module enable idm:DL1

      4. sudo dnf install freeipa-server ipa-server-dns bind-dyndb-ldap -y

  7. Run the following command to install the LDAP replica
  8. ipa-replica-install --domain "eumetsat.sandbox.ewc" --principal sandbox-ldap-admin --admin-password hunter2 --server <OLD LDAP complete domain (e.g. ldap.eumetsat.sandbox.ewc)> --setup-ca --setup-dns --force-join --forwarder=8.8.8.8 --forwarder=1.1.1.1 --no-host-dns

    (add --verbose for more logs) NOTE: go ahead with the default netbios and also put yes when asked about the ipa-sidgen task installation for users

    --no-host-dns         Do not use DNS for hostname lookup during installation


Migrate from Rocky 8 to Rocky 9

rocky 9

Creating a new LDAP to migrate to

  1. deploy a new machine with
    1. rocky 8
    2. new ldap security group
    3. plan eo1.medium
  2. become sudo
  3. stop and disable firewalld on the machine systemctl stop firewalld && systemctl disable firewalld
  4. add old ldap IP and domain of the machine acting as LDAP in one line to /etc/hosts on this machine
  5. [test234@test-podman2 ~]$ cat /etc/hosts
<!-- BEGIN ANSIBLE MANAGED BLOCK -->
10.0.0.133 test-podman2.eumetsat.sandbox.ewc
10.0.0.63 <OLD LDAP complete domain (e.g. ldap.eumetsat.sandbox.ewc)>
<!-- END ANSIBLE MANAGED BLOCK -->
  6. Install dependencies
    1. for rocky 9: Install ipa-replica-install command (dnf install ipa-server ipa-server-dns -y) (second is required because it is required for the option in the ipa-replica-install)
  7. Run the following command to install the LDAP replica
  8. ipa-replica-install --domain "eumetsat.sandbox.ewc" --principal sandbox-ldap-admin --admin-password hunter2 --server <OLD LDAP complete domain (e.g. ldap.eumetsat.sandbox.ewc)> --setup-ca --setup-dns --force-join --forwarder=8.8.8.8 --forwarder=1.1.1.1 --no-host-dns

    add --verbose for more logs


9. ipa dnszone-mod <DNS ZONE you find with ipa dnszone-find> --name-server=<NEWLDAP complete domain (e.g. ldap-test-rocky.eumetsat.sandbox.ewc)>

  1. on the new ldap machine specify which ldap to be master (using command: ipa-csreplica-manage set-renewal-master)

  2. DON'T DO IT, see next steps comment on why → create an A record for existing ldap domain to point to the new LDAP ipa dnsrecord-add eumetsat.sandbox.ewc. ldap --a-rec <NEW_LDAP_IP> 

  3. Modify the DNS primary in Network (cannot be done by a user, unless we give permissions to modify networks) and wait some time (1 week? ) or do manual action on each machine to make the changes: (lease time ~5.20 hours)
    1. for Ubuntu a restart of the machine or the newotk systemctl restart networking or maybe after while you will get it (TBT)
    2. for Rocky 9 network manager has priority on resolve.conf
    3. for Rocky 8 happening in sandbox but not in mcat, one option either remove /etc/resolve.conf and dhclient or ??
  4. Change the secret/ldap_hostname to point to the new ipa host

  5. sudo ipactl stop on the old LDAP machine (check if this step can be removed on another test run)

  6. ipa-replica-manage del ldap.eumetsat.sandbox.ewc --force delete the replica ldap


TBD

We've found a problem after the IPA Migration:

ipa: ERROR: Operations error: Allocation of a new value for range cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config failed! Unable to proceed.


There is a FreeIPA doc site related to this: https://www.freeipa.org/page/V3/Recover_DNA_Ranges.html
I've fixed this using ipa-replica-manage dnarange-set $ldap_server ${min}-${max}
I've found the range we used with ipa idrange-find and set the lower boundary to exclude every existing account (as of ipa user-find | grep 'UID')

I'm not sure, why it didn't do that automatically (as I understand the documentation, that is what should have happened), but that might be because of the age of the old server or because the old server blocked every valid range...
To check the range that is used: ipa-replica-manage dnarange-show which could be used during migration.

There is a second command pair (dnanextrange-set & dnanextrange-show), where I'm not sure what it does, I didn't set it, and it still worked. Just to let you know, might be good to set during migration anyway.



Run workflow to switch IP interfaces between LDAP

alternative path from 9.

10. detach the interface from the old LDAP 

11 update the ip in the /etc/hosts and remove the old LDAP record

12. Delete old LDAP DNS records 

ipa-replica-manage del ldap.eumetsat.sandbox.ewc --force

13. replace the new ldap IP with the IP of the interface of the old LDAP →

ipa dnsrecord-mod <hostedzone> ipa-ca --a-rec <OLD_LDAP_IP>
ipa dnsrecord-mod <DNS_ZONE> <NEW LDAP HOSTNAME>--a-rec <OLD_LDAP_IP>

14. switch off the new LDAP

15. remove interface 

16. add interface with the IP of the old LDAP

17. add security groups to new LDAP

18. restart new LDAP machine



Tests

19 login with DNS

20 ipactl status

21 change the value in Cypher: Change the secret/ldap_hostname to point to the new ipa host

22 deploy a new machine and enroll


v 4.6.8 default 

Procedure for new LDAPs

  • Modify the ldap base image to be rocky 9

LDAP machine

  1. check if reverse zone exists and add dns reverse into deployment
  2. populate it for all existing machines in the tenancy (needs script dumping the forward zone)
  • include reverse zone flag (nope, we have this and it seems not to work for private IP ranges, we have to do it manually) → create manually the reverse zone in the workflow - probably just add ipa dnszone-add --name-from-ip=192.0.2.0/24 to the ansible


enrolling workflow (to be double checked

  1. we modify the enroll script to include --subids and we add an extra line at the end of the default workflow for machines to run sudo rpm --restore shadow-utils
  2. Remove /etc/NetworkManager/conf.d/99-cloud-init.conf


Write a script that verifies whether a machine is gone (how? no ping? for a long time?)  and destroys ldap entries relating to it.  Put this in the crontab on all ldaps → now with openstack Application Credentials it can be done!!



PODMAN Installation issue with subids

https://freeipa.readthedocs.io/en/latest/designs/subordinate-ids.html#ipa-plugins-and-commands

on LDAP Server

Generate subids for existing users:
sudo /usr/libexec/ipa/ipa-subids --group ipausers

Enable auto generate for new users:
ipa config-mod --user-default-subid=true



Known errors

on centos7 version 4.6.8 tryting to move to rocky 9 you will get errors, you have to move step by step with the OS versions. You can see some of the errors and possible workaround but it won't work as of today tests done. Better to follow the approach above, moving from centos7 to rocky8 and from rocky8 to rocky9.


first error: 

  [4/30]: creating installation admin user
Unable to log in as uid=admin-new-ldap.datapro.ewc,ou=people,o=ipaca on ldap://ldap.datapro.ewc:389
[hint] tune with replication_wait_timeout
  [error] NotFound: uid=admin-new-ldap.datapro.ewc,ou=people,o=ipaca did not replicate to ldap://ldap.datapro.ewc:389
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

uid=admin-new-ldap.datapro.ewc,ou=people,o=ipaca did not replicate to ldap://ldap.datapro.ewc:389
The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

MORE LOGS:
2024-10-18T08:59:21Z DEBUG Waiting 300 seconds for uid=admin-new-ldap.datapro.ewc,ou=people,o=ipaca to appear on ldap://ldap.datapro.ewc:389
2024-10-18T09:04:22Z ERROR Unable to log in as uid=admin-new-ldap.datapro.ewc,ou=people,o=ipaca on ldap://ldap.datapro.ewc:389
2024-10-18T09:04:22Z INFO [hint] tune with replication_wait_timeout
2024-10-18T09:04:22Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 686, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 672, in run_step
    method()
  File "/usr/lib/python3.9/site-packages/ipaserver/install/dogtaginstance.py", line 789, in setup_admin
    raise errors.NotFound(
ipalib.errors.NotFound: uid=admin-new-ldap.datapro.ewc,ou=people,o=ipaca did not replicate to ldap://ldap.datapro.ewc:389

2024-10-18T09:04:22Z DEBUG   [error] NotFound: uid=admin-new-ldap.datapro.ewc,ou=people,o=ipaca did not replicate to ldap://ldap.datapro.ewc:389
2024-10-18T09:04:22Z DEBUG Removing /root/.dogtag/pki-tomcat/ca
2024-10-18T09:04:22Z DEBUG   File "/usr/lib/python3.9/site-packages/ipapython/admintool.py", line 180, in execute
    return_value = self.run()
  File "/usr/lib/python3.9/site-packages/ipapython/install/cli.py", line 344, in run
    return cfgr.run()
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 360, in run
    return self.execute()
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 386, in execute
    for rval in self._executor():
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 435, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 468, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 458, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 425, in __runner
    step()
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 419, in step_next
    return next(self.__gen)
  File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 663, in _configure
    next(executor)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 435, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 468, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 526, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 458, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 523, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 458, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 425, in __runner
    step()
  File "/usr/lib/python3.9/site-packages/ipapython/install/core.py", line 419, in step_next
    return next(self.__gen)
  File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.9/site-packages/ipapython/install/common.py", line 65, in _install
    for unused in self._installer(self.parent):
  File "/usr/lib/python3.9/site-packages/ipaserver/install/server/__init__.py", line 599, in main
    replica_install(self)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/server/replicainstall.py", line 401, in decorated
    func(installer)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/server/replicainstall.py", line 1392, in install
    ca.install(False, config, options, custodia=custodia)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/ca.py", line 354, in install
    install_step_0(standalone, replica_config, options, custodia=custodia)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/ca.py", line 423, in install_step_0
    ca.configure_instance(
  File "/usr/lib/python3.9/site-packages/ipaserver/install/cainstance.py", line 505, in configure_instance
    self.start_creation(runtime=runtime)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 686, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 672, in run_step
    method()
  File "/usr/lib/python3.9/site-packages/ipaserver/install/dogtaginstance.py", line 789, in setup_admin
    raise errors.NotFound(

2024-10-18T09:04:22Z DEBUG The ipa-replica-install command failed, exception: NotFound: uid=admin-new-ldap.datapro.ewc,ou=people,o=ipaca did not replicate to ldap://ldap.datapro.ewc:389
2024-10-18T09:04:22Z ERROR uid=admin-new-ldap.datapro.ewc,ou=people,o=ipaca did not replicate to ldap://ldap.datapro.ewc:389
2024-10-18T09:04:22Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

solution:

current ldap are on 3.6.8 with python 2.7 so the bug above needs to be changed on this file: /usr/lib/python2.7/site-packages/ipaserver/secrets/store.py in PEMfilehandler class for export_key method.  (Line 211)


        '-keypbe', 'AES-256-CBC',
        '-certpbe', 'AES-256-CBC',
        '-macalg', 'sha384',


on centos7 version 4.6.8 tried updating libraries, but the dns broke and still not fixed:

[root@ldap murdaca]# systemctl status -l named-pkcs11.service
● named-pkcs11.service - Berkeley Internet Name Domain (DNS) with native PKCS#11
   Loaded: loaded (/usr/lib/systemd/system/named-pkcs11.service; disabled; vendor preset: disabled)
   Active: failed (Result: signal) since Thu 2024-10-17 15:24:59 UTC; 13h ago
  Process: 11930 ExecStart=/usr/sbin/named-pkcs11 -u named -c ${NAMEDCONF} $OPTIONS (code=exited, status=0/SUCCESS)
  Process: 11927 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z "$NAMEDCONF"; else echo "Checking of zone files is disabled"; fi (code=exited, status=0/SUCCESS)
 Main PID: 11932 (code=killed, signal=ABRT)

Oct 17 15:24:59 ldap.batchpro.ewc named-pkcs11[11932]: #6 0x7f26fc529b89 in ??
Oct 17 15:24:59 ldap.batchpro.ewc named-pkcs11[11932]: #7 0x7f26fc533358 in ??
Oct 17 15:24:59 ldap.batchpro.ewc named-pkcs11[11932]: #8 0x7f270b9aa713 in ??
Oct 17 15:24:59 ldap.batchpro.ewc named-pkcs11[11932]: #9 0x7f270b9ab28b in ??
Oct 17 15:24:59 ldap.batchpro.ewc named-pkcs11[11932]: #10 0x7f2709a80ea5 in ??
Oct 17 15:24:59 ldap.batchpro.ewc named-pkcs11[11932]: #11 0x7f2708af3b0d in ??
Oct 17 15:24:59 ldap.batchpro.ewc named-pkcs11[11932]: exiting (due to assertion failure)
Oct 17 15:24:59 ldap.batchpro.ewc systemd[1]: named-pkcs11.service: main process exited, code=killed, status=6/ABRT
Oct 17 15:24:59 ldap.batchpro.ewc systemd[1]: Unit named-pkcs11.service entered failed state.
Oct 17 15:24:59 ldap.batchpro.ewc systemd[1]: named-pkcs11.service failed.

check all logs at /var/named/data/named.run 

18-Oct-2024 06:17:54.898 managed-keys-zone: loaded serial 734
18-Oct-2024 06:17:54.898 zone 0.in-addr.arpa/IN: loaded serial 0
18-Oct-2024 06:17:54.899 zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0
18-Oct-2024 06:17:54.900 zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0
18-Oct-2024 06:17:54.901 zone localhost/IN: loaded serial 0
18-Oct-2024 06:17:54.902 zone localhost.localdomain/IN: loaded serial 0
18-Oct-2024 06:17:54.902 all zones loaded
18-Oct-2024 06:17:54.902 running
18-Oct-2024 06:17:54.903 ../../../lib/dns-pkcs11/name.c:1114: REQUIRE((target != ((void *)0) && (__builtin_expect(!!((target) != ((void *)0)), 1) && __builtin_expect(!!(((const isc__magic_t *)(target))->magic == (0x42756621U)), 1))) || (target == ((void *)0) && (__builtin_expect(!!((name->buffer) != ((void *)0)), 1) && __builtin_expect(!!(((const isc__magic_t *)(name->buffer))->magic == (0x42756621U)), 1)))) failed, back trace
18-Oct-2024 06:17:54.903 #0 0x564390897130 in ??
18-Oct-2024 06:17:54.903 #1 0x7fcdba9d048a in ??
18-Oct-2024 06:17:54.903 #2 0x7fcdbacd5b7b in ??
18-Oct-2024 06:17:54.903 #3 0x7fcda7540b78 in ??
18-Oct-2024 06:17:54.903 #4 0x7fcda7540fee in ??
18-Oct-2024 06:17:54.903 #5 0x7fcda754298b in ??
18-Oct-2024 06:17:54.903 #6 0x7fcda7542b89 in ??
18-Oct-2024 06:17:54.903 #7 0x7fcda754c358 in ??
18-Oct-2024 06:17:54.903 #8 0x7fcdba9f5713 in ??
18-Oct-2024 06:17:54.903 #9 0x7fcdba9f628b in ??
18-Oct-2024 06:17:54.903 #10 0x7fcdb8acbea5 in ??
18-Oct-2024 06:17:54.903 #11 0x7fcdb7b3eb0d in ??
18-Oct-2024 06:17:54.903 exiting (due to assertion failure)





  • No labels