NiFi: Kerberize/SSL

(Last Updated On: )

In this tutorial I will show you how to use Kerberos/SSL with NiFi. I will use self signed certs for this example. Before you begin ensure you have installed Kerberos Server and NiFi.

This assumes your hostname is “hadoop”

Create Kerberos Principals

  1. cd /etc/security/keytabs/
  2.  
  3. sudo kadmin.local
  4.  
  5. #You can list principals
  6. listprincs
  7.  
  8. #Create the following principals
  9. addprinc -randkey nifi/hadoop@REALM.CA
  10. addprinc -randkey nifi-spnego/hadoop@REALM.CA
  11. #Notice this user does not have -randkey because we are a login user
  12. #Also notice that this user does not have a keytab created
  13. addprinc admin/hadoop@REALM.CA
  14.  
  15.  
  16. #Create the keytab files.
  17. #You will need these for Hadoop to be able to login
  18. xst -k nifi.service.keytab nifi/hadoop@REALM.CA
  19. xst -k nifi-spnego.service.keytab nifi-spnego/hadoop@REALM.CA

Set Keytab Permissions/Ownership

  1. sudo chown root:hadoopuser /etc/security/keytabs/*
  2. sudo chmod 750 /etc/security/keytabs/*

Stop NiFi

  1. sudo service nifi stop

Hosts Update

  1. sudo nano /etc/hosts
  2.  
  3. #Remove 127.0.1.1 line
  4.  
  5. #Change 127.0.0.1 to the following
  6. 127.0.0.1 gaudreault_kdc.ca hadoop localhost

Ubuntu Firewall

  1. sudo ufw disable

sysctl.conf

Disable ipv6 as it causes issues in getting your server up and running.

  1. nano /etc/sysctl.conf

Add the following to the end and save

  1. net.ipv6.conf.all.disable_ipv6 = 1
  2. net.ipv6.conf.default.disable_ipv6 = 1
  3. net.ipv6.conf.lo.disable_ipv6 = 1
  4. #Change eth0 to what ifconfig has
  5. net.ipv6.conf.eth0.disable_ipv6 = 1

Close sysctl

  1. sysctl -p
  2. cat /proc/sys/net/ipv6/conf/all/disable_ipv6
  3. reboot

TrustStore / KeyStore

  1. #Creating your Certificate Authority
  2. sudo mkdir -p /etc/security/serverKeys
  3. sudo chown -R root:hduser /etc/security/serverKeys/
  4. sudo chmod 750 /etc/security/serverKeys/
  5.  
  6. cd /etc/security/serverKeys
  7.  
  8. sudo openssl genrsa -aes128 -out nifi.key 4096
  9. sudo openssl req -x509 -new -key nifi.key -days 1095 -out nifi.pem
  10. sudo openssl rsa -check -in nifi.key #check it
  11. sudo openssl x509 -outform der -in nifi.pem -out nifi.der
  12. sudo keytool -import -keystore truststore.jks -file nifi.der -alias nifi
  13. #***You must type 'yes' to trust this certificate.
  14. sudo keytool -v -list -keystore truststore.jks
  15.  
  16. #Creating your Server Keystore
  17. sudo keytool -genkey -alias nifi -keyalg RSA -keystore keystore.jks -keysize 2048
  18. sudo keytool -certreq -alias nifi -keystore keystore.jks -file nifi.csr
  19. sudo openssl x509 -sha256 -req -in nifi.csr -CA nifi.pem -CAkey nifi.key -CAcreateserial -out nifi.crt -days 730
  20. sudo keytool -import -keystore keystore.jks -file nifi.pem
  21. sudo keytool -import -trustcacerts -alias nifi -file nifi.crt -keystore keystore.jks
  22.  
  23. sudo chown -R root:hduser /etc/security/serverKeys/*
  24. sudo chmod 750 /etc/security/serverKeys/*

nifi.properties

  1. cd /usr/local/nifi/conf/
  2. nano nifi.properties
  3.  
  4. #Find "# Site to Site properties" and change the following properties to what is below
  5.  
  6. nifi.remote.input.host=
  7. nifi.remote.input.secure=true
  8. nifi.remote.input.socket.port=9096
  9. nifi.remote.input.http.enabled=false
  10.  
  11. #Find "# web properties #" and change the following properties to what is below
  12.  
  13. nifi.web.http.host=
  14. nifi.web.http.port=
  15. nifi.web.https.host=0.0.0.0
  16. nifi.web.https.port=9095
  17.  
  18. #Find "# security properties #" and change the following properties to what is below
  19.  
  20. nifi.security.keystore=/etc/security/serverKeys/keystore.jks
  21. nifi.security.keystoreType=JKS
  22. nifi.security.keystorePasswd=PASSWORD
  23. nifi.security.keyPasswd=PASSWORD
  24. nifi.security.truststore=/etc/security/serverKeys/truststore.jks
  25. nifi.security.truststoreType=JKS
  26. nifi.security.truststorePasswd=PASSWORD
  27. nifi.security.needClientAuth=true
  28. nifi.security.user.authorizer=managed-authorizer
  29. nifi.security.user.login.identity.provider=kerberos-provider
  30.  
  31. #Find "# Core Properties #" and change the following properties to what is below
  32.  
  33. nifi.authorizer.configuration.file=./conf/authorizers.xml
  34. nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
  35.  
  36. #Find "# kerberos #" and change the following properties to what is below
  37.  
  38. nifi.kerberos.krb5.file=/etc/krb5.conf
  39.  
  40. #Find "# kerberos service principal #" and change the following properties to what is below
  41.  
  42. nifi.kerberos.service.principal=nifi/hadoop@REALM.CA
  43. nifi.kerberos.service.keytab.location=/etc/security/keytabs/nifi.service.keytab
  44.  
  45. #Find "# kerberos spnego principal #" and change the following properties to what is below
  46.  
  47. nifi.kerberos.spnego.principal=nifi-spnego/hadoop@REALM.CA
  48. nifi.kerberos.spnego.keytab.location=/etc/security/keytabs/nifi-spnego.service.keytab
  49. nifi.kerberos.spnego.authentication.expiration=12 hours
  50.  
  51. #Find "# cluster common properties (all nodes must have same values) #" and change the following properties to what is below
  52.  
  53. nifi.cluster.protocol.is.secure=true

login-identity-providers.xml

  1. nano login-identity-providers.xml
  2.  
  3. #Find "kerberos-provider"
  4. <provider>
  5. 	<identifier>kerberos-provider</identifier>
  6. 	<class>org.apache.nifi.kerberos.KerberosProvider</class>
  7. 	<property name="Default Realm">REALM.CA</property>
  8. 	<property name="Kerberos Config File">/etc/krb5.conf</property>
  9. 	<property name="Authentication Expiration">12 hours</property>
  10. </provider>

authorizers.xml

  1. nano authorizers.xml
  2.  
  3. #Find "file-provider"
  4. <authorizer>
  5. 	<identifier>file-provider</identifier>
  6. 	<class>org.apache.nifi.authorization.FileAuthorizer</class>
  7. 	<property name="Authorizations File">./conf/authorizations.xml</property>
  8. 	<property name="Users File">./conf/users.xml</property>
  9. 	<property name="Initial Admin Identity">admin/hadoop@REALM.CA</property>
  10. 	<property name="Legacy Authorized Users File"></property>
  11.  
  12. 	<property name="Node Identity 1"></property>
  13. </authorizer>

Start Nifi

  1. sudo service nifi start

NiFi Web Login

Issues:

  • If you get the error “No applicable policies could be found” after logging in and no GUI is shown stop the NiFi service and restart. Then you should be good.
  • If you can then login but you don’t have any policies still you will need to update “authorizations.xml” and add the below lines. Making sure to change the resource process group id to the root process group id and the user id to the user id
  1. nano /usr/local/nifi/conf/authorizations.xml
  2.  
  3. <policy identifier="1c897e9d-3dd5-34ca-ae3d-75fb5ee3e1a5" resource="/data/process-groups/##CHANGE TO ROOT ID##" action="R">
  4. 	<user identifier="##CHANGE TO USER ID##"/>
  5. </policy>
  6. <policy identifier="91c64c2d-7848-371d-9d5f-db71138b152f" resource="/data/process-groups/##CHANGE TO ROOT ID##" action="W">
  7. 	<user identifier="##CHANGE TO USER ID##"/>
  8. </policy>
  9. <policy identifier="7aeb4d67-e2e1-3a3e-a8fa-94576f35539e" resource="/process-groups/##CHANGE TO ROOT ID##" action="R">
  10. 	<user identifier="##CHANGE TO USER ID##"/>
  11. </policy>
  12. <policy identifier="f5b620e0-b094-3f70-9542-dd6920ad5bd9" resource="/process-groups/##CHANGE TO ROOT ID##" action="W">
  13. 	<user identifier="##CHANGE TO USER ID##"/>
  14. </policy>

References

https://community.hortonworks.com/articles/34147/nifi-security-user-authentication-with-kerberos.html

https://community.hortonworks.com/content/supportkb/151106/nifi-how-to-create-your-own-certs-for-securing-nif.html