Apple Resource Programming Guide Manuel Apple sur Fnac.com - Pour voir la liste complète des manuels APPLE, cliquez ici

 

 

TELECHARGER LE PDF sur :

https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/LoadingResources/LoadingResources.pdf

Commander un produit Apple sur Fnac.com

 

 

Voir également d'autres Guides APPLE :

Apple-macbook_pro-retina-mid-2012-important_product_info_f.pdf-manuel

Apple-iOS_Security_May12.pdf-manue

Apple-Mac-Pro-2008-Performance-and-Productivity-for-Creative-Pros

Apple-iPod_shuffle_4thgen_Manuale_utente.pdf-Italie-Manuel

Apple-KernelProgramming.pdf-manuel

Apple-Core-Data-Model-Versioning-and-Data-Migration-Programming-Guide-manuel

Apple-RED_Workflows_with_Final_Cut_Pro_X.pdf-manuel

Apple-Transitioning-to-ARC-Release-Notes-manuel

Apple-iTunes-Connect-Sales-and-Trends-Guide-manuel

Apple-App-Sandbox-Design-Guide-manuel

Apple-String-Programming-Guide-manuel

Apple-Secure-Coding-Guide-manuel

Apple_AirPort_Networks_Early2009.pdf-manuel

Apple-TimeCapsule_SetupGuide_TA.pdf-manuel

Apple-time_capsule_4th_gen_setup.pdf-manuel

Apple-TimeCapsule_SetupGuide.pdf-manuel

Apple-TimeCapsule_SetupGuide_CH.pdf-Chinois-manuel

Apple-CodeSigningGuide.pdf-manuel

Apple-ViewControllerPGforiOS.pdf-manuel

Apple-KeyValueObserving.pdf-manuel

Apple-mac_mini-late-2012-quick_start.pdf-manuel

Apple-OS-X-Mountain-Lion-Core-Technologies-Overview-June-2012-manuel

Apple-OS-X-Server-Product-Overview-June-2012-manuel

Apple-Apple_Server_Diagnostics_UG_109.pdf-manuel

Apple-PackageMaker_UserGuide.pdf-manuel

Apple-Instrumentos_y_efectos_de_Logic_Studio.pdf-Manuel

Apple-ipod_nano_kayttoopas.pdf-Finlande-Manuel

Apple_ProRes_White_Paper_October_2012.pdf-Manuel

Apple-wp_osx_configuration_profiles.pdf-Manuel

Apple-UsingiTunesProducerFreeBooks.pdf-Manuel

Apple-ipad_manual_do_usuario.pdf-Portugais-Manuel

Apple-Instruments_et_effets_Logic_Studio.pdf-Manuel

Apple-ipod_touch_gebruikershandleiding.pdf-Neerlandais-Manuel

AppleiPod_shuffle_4thgen_Manual_del_usuario.pdf-Espagnol-Manuel

Apple-Premiers-contacts-avec-votre-PowerBook-G4-Manuel

Apple_Composite_AV_Cable.pdf-Manuel

Apple-iPod_shuffle_3rdGen_UG_DK.pdf-Danemark-Manuel

Apple-iPod_classic_160GB_Benutzerhandbuch.pdf-Allemand-Manuel

Apple-VoiceOver_GettingStarted-Manuel

Apple-iPod_touch_2.2_Benutzerhandbuch.pdf-Allemand-Manuel

Apple-Apple_TV_Opstillingsvejledning.pdf-Allemand-Manuel

Apple-iPod_shuffle_4thgen_Manuale_utente.pdf-Italie-Manuel

Apple-iphone_prirucka_uzivatela.pdf-Manuel

Apple-Aan-de-slag-Neerlandais-Manuel

Apple-airmac_express-80211n-2nd-gen_setup_guide.pdf-Thailande-Manuel

Apple-ipod_nano_benutzerhandbuch.pdf-Allemand-Manuel

Apple-aperture3.4_101.pdf-Manuel

Apple-Pages09_Anvandarhandbok.pdf-Manuel

Apple-nike_plus_ipod_sensor_ug_la.pdf-Mexique-Manuel

Apple-ResEdit-Reference-For-ResEdit02.1-Manuel

Apple-ipad_guide_de_l_utilisateur.pdf-Manuel

Apple-Compressor-4-Benutzerhandbuch-Allemand-Manuel

Apple-AirPort_Networks_Early2009_DK.pdf-Danemark-Manuel

Apple-MacBook_Pro_Mid2007_2.4_2.2GHz_F.pdf-Manuel

Apple-MacBook_13inch_Mid2010_UG_F.pdf-Manuel

Apple-Xserve-RAID-Presentation-technologique-Janvier-2004-Manuel

Apple-MacBook_Pro_15inch_Mid2010_F.pdf-Manuel

Apple-AirPort_Express-opstillingsvejledning.pdf-Danemark-Manuel

Apple-DEiPod_photo_Benutzerhandbuch_DE0190269.pdf-Allemand-Manuel

Apple-Final-Cut-Pro-X-Logic-Effects-Reference-Manuel

Apple-iPod_touch_2.1_Brugerhandbog.pdf-Danemark-Manuel

Apple-Remote-Desktop-Administratorhandbuch-Version-3.1-Allemand-Manuel

Apple-Qmaster-4-User-Manual-Manuel

Apple-Server_Administration_v10.5.pdf-Manuel

Apple-ipod_classic_features_guide.pdf-Manuel

Apple-Lecteur-Optique-Manuel

Apple-Carte-AirPort-Manuel

Apple-iPhone_Finger_Tips_Guide.pdf-Anglais-Manuel

Apple-Couvercle-Manuel

Apple-battery.cube.pdf-Manuel

Apple-Boitier-de-l-ordinateur-Manuel

Apple-Pile-Interne-Manuel

Apple-atacable.pdf-Manuel

Apple-videocard.pdf-Manuel

Apple-Guide_de_configuration_de_l_Airport_Express_5.1.pdf-Manuel

Apple-iMac_Mid2010_UG_F.pdf-Manuel

Apple-MacBook_13inch_Mid2009_F.pdf-Manuel

Apple-MacBook_Mid2007_UserGuide.F.pdf-Manuel

Apple-Designing_AirPort_Networks_10.5-Windows_F.pdf-Manuel

Apple-Administration_de_QuickTime_Streaming_et_Broadcasting_10.5.pdf-Manuel

Apple-Opstillingsvejledning_til_TimeCapsule.pdf-Danemark-Manuel

Apple-iPod_nano_5th_gen_Benutzerhandbuch.pdf-Manuel

Apple-iOS_Business.pdf-Manuel

Apple-AirPort_Extreme_Installationshandbuch.pdf-Manuel

Apple-Final_Cut_Express_4_Installation_de_votre_logiciel.pdf-Manuel

Apple-MacBook_Pro_15inch_2.53GHz_Mid2009.pdf-Manuel

Apple-Network_Services.pdf-Manuel

Apple-Aperture_Performing_Adjustments_f.pdf-Manuel

Apple-Supplement_au_guide_Premiers_contacts.pdf-Manuel

Apple-Administration_des_images_systeme_et_de_la_mise_a_jour_de_logiciels_10.5.pdf-Manuel

Apple-Mac_OSX_Server_v10.6_Premiers_contacts.pdf-Francais-Manuel

Apple-Designing_AirPort_Networks_10.5-Windows_F.pdf-Manuel

Apple-Mise_a_niveau_et_migration_v10.5.pdf-Manue

Apple-MacBookPro_Late_2007_2.4_2.2GHz_F.pdf-Manuel

Apple-Mac_mini_Late2009_SL_Server_F.pdf-Manuel

Apple-Mac_OS_X_Server_10.5_Premiers_contacts.pdf-Manuel

Apple-iPod_touch_2.0_Guide_de_l_utilisateur_CA.pdf-Manuel

Apple-MacBook_Pro_17inch_Mid2010_F.pdf-Manuel

Apple-Comment_demarrer_Leopard.pdf-Manuel

Apple-iPod_2ndGen_USB_Power_Adapter-FR.pdf-Manuel

Apple-Feuille_de_operations_10.4.pdf-Manuel

Apple-Time_Capsule_Installationshandbuch.pdf-Allemand-Manuel

Apple-F034-2262AXerve-grappe.pdf-Manuel

Apple-Mac_Pro_Early2009_4707_UG_F

Apple-imacg5_17inch_Power_Supply

Apple-Logic_Studio_Installieren_Ihrer_Software_Retail

Apple-IntroductionXserve1.0.1

Apple-Aperture_Getting_Started_d.pdf-Allemand

Apple-getting_started_with_passbook

Apple-iPod_mini_2nd_Gen_UserGuide.pdf-Anglais

Apple-Deploiement-d-iPhone-et-d-iPad-Reseaux-prives-virtuels

Apple-F034-2262AXerve-grappe

Apple-Mac_OS_X_Server_Glossaire_10.5

Apple-FRLogic_Pro_7_Guide_TDM

Apple-iphone_bluetooth_headset_userguide

Apple-Administration_des_services_reseau_10.5

Apple-imacg5_17inch_harddrive

Apple-iPod_nano_4th_gen_Manuale_utente

Apple-iBook-G4-Getting-Started

Apple-XsanGettingStarted

Apple-Mac_mini_UG-Early2006

Apple-Guide_des_fonctionnalites_de_l_iPod_classic

Apple-Guide_de_configuration_d_Xsan_2

Apple-MacBook_Late2006_UsersGuide

Apple-sur-Fnac.com

Apple-Mac_mini_Mid2010_User_Guide_F.pdf-Francais

Apple-PowerBookG3UserManual.PDF.Anglais

Apple-Installation_de_votre_logiciel_Logic_Studio_Retail

Apple-Pages-Guide-de-l-utilisateur

Apple-MacBook_Pro_13inch_Mid2009.pdf.Anglais

Apple-MacBook_Pro_15inch_Mid2009

Apple-Installation_de_votre_logiciel_Logic_Studio_Upgrade

Apple-FRLogic_Pro_7_Guide_TDM

Apple-airportextreme_802.11n_userguide

Apple-iPod_shuffle_3rdGen_UG

Apple-iPod_classic_160GB_User_Guide

Apple-iPod_nano_5th_gen_UserGuide

Apple-ipod_touch_features_guide

Apple-Wireless_Mighty_Mouse_UG

Apple-Advanced-Memory-Management-Programming-Guide

Apple-iOS-App-Programming-Guide

Apple-Concurrency-Programming-Guide

Apple-MainStage-2-User-Manual-Anglais

Apple-iMacG3_2002MultilingualUserGuide

Apple-iBookG3_DualUSBUserGuideMultilingual.PDF.Anglais

Apple-imacG5_20inch_AirPort

Apple-Guide_de_l_utilisateur_de_Mac_Pro_Early_2008

Apple-Installation_de_votre_logiciel_Logic_Express_8

Apple-iMac_Guide_de_l_utilisateur_Mid2007

Apple-imacg5_20inch_OpticalDrive

Apple-FCP6_Formats_de_diffusion_et_formats_HD

Apple-prise_en_charge_des_surfaces_de_controle_logic_pro_8

Apple-Aperture_Quick_Reference_f

Apple-Shake_4_User_Manual

Apple-aluminumAppleKeyboard_wireless2007_UserGuide

Apple-ipod_shuffle_features_guide

Apple-Color-User-Manual

Apple-XsanGettingStarted

Apple-Migration_10.4_2e_Ed

Apple-MacBook_Air_SuperDrive

Apple-MacBook_Late2007-f

ApplePowerMacG5_(Early_2005)_UserGuide

Apple-iSightUserGuide

Apple-MacBook_Pro_Early_2008_Guide_de_l_utilisateur

Apple-Nouvelles-fonctionnalites-aperture-1.5

Apple-premiers_contacts_2e_ed_10.4.pdf-Mac-OS-X-Server

Apple-premiers_contacts_2e_ed_10.4

Apple-eMac_2005UserGuide

Apple-imacg5_20inch_Inverter

Apple-Keynote2_UserGuide.pdf-Japon

Apple-Welcome_to_Tiger.pdf-Japon

Apple-XsanAdminGuide_j.pdf-Japon

Apple-PowerBookG4_UG_15GE.PDF-Japon

Apple-Xsan_Migration.pdf-Japon

Apple-Xserve_Intel_DIY_TopCover_JA.pdf-Japon

Apple-iPod_nano_6thgen_User_Guide_J.pdf-Japon

Apple-Aperture_Photography_Fundamentals.pdf-Japon

Apple-nikeipod_users_guide.pdf-Japon

Apple-QuickTime71_UsersGuide.pdf-Japon

Apple-iMacG5_iSight_UG.pdf-Japon

Apple-Aperture_Performing_Adjustments_j.pdf-Japon

Apple-iMacG5_17inch_HardDrive.pdf-Japon

Apple-iPod_shuffle_Features_Guide_J.pdf-Japon

Apple-MacBook_Air_User_Guide.pdf-Japon

Apple-MacBook_UsersGuide.pdf-Japon

Apple-iPad_iOS4_Brukerhandbok.pdf-Norge-Norvege

Apple-Apple_AirPort_Networks_Early2009_H.pd-Norge-Norvege

Apple-iPod_classic_120GB_no.pdf-Norge-Norvege

Apple-StoreKitGuide.pdf-Japon

Apple-Xserve_Intel_DIY_ExpansionCardRiser_JA.pdf-Japon

Apple-iMacG5_Battery.pdf-Japon

Apple-Logic_Pro_8_Getting_Started.pdf-Japon

Apple-PowerBook-handbok-Norge-Norveg

Apple-iWork09_formler_og_funksjoner.pdf-Norge-Norvege

Apple-MacBook_Pro_15inch_Mid2010_H.pdf-Norge-Norvege

Apple-MacPro_HardDrive_DIY.pdf-Japon

Apple-iPod_Fifth_Gen_Funksjonsoversikt.pdf-Norge-Norvege

Apple-MacBook_13inch_white_Early2009_H.pdf-Norge-Norvege

Apple-GarageBand_09_Komme_i_gang.pdf-Norge-Norvege

Apple-MacBook_Pro_15inch_Mid2009_H.pdf-Norge-Norvege

Apple-imac_mid2011_ug_h.pdf-Norge-Norvege

Apple-iDVD_08_Komme_i_gang.pdf-Norge-Norvege

Apple-MacBook_Air_11inch_Late2010_UG_H.pdf-Norge-Norvege

Apple-iMac_Mid2010_UG_H.pdf-Norge-Norvege

Apple-MacBook_13inch_Mid2009_H.pdf-Norge-Norvege

/Apple-iPhone_3G_Viktig_produktinformasjon_H-Norge-Norvege

Apple-MacBook_13inch_Mid2010_UG_H.pdf-Norge-Norvege

Apple-macbook_air_13inch_mid2011_ug_no.pdf-Norge-Norvege

Apple-Mac_mini_Early2009_UG_H.pdf-Norge-Norvege

Apple-ipad2_brukerhandbok.pdf-Norge-Norvege

Apple-iPhoto_08_Komme_i_gang.pdf-Norge-Norvege

Apple-MacBook_Air_Brukerhandbok_Late2008.pdf-Norge-Norvege

Apple-Pages09_Brukerhandbok.pdf-Norge-Norvege

Apple-MacBook_13inch_Late2009_UG_H.pdf-Norge-Norvege

Apple-iPhone_3GS_Viktig_produktinformasjon.pdf-Norge-Norvege

Apple-MacBook_13inch_Aluminum_Late2008_H.pdf-Norge-Norvege

Apple-Wireless_Keyboard_Aluminum_2007_H-Norge-Norvege

Apple-NiPod_photo_Brukerhandbok_N0190269.pdf-Norge-Norvege

Apple-MacBook_Pro_13inch_Mid2010_H.pdf-Norge-Norvege

Apple-MacBook_Pro_17inch_Mid2010_H.pdf-Norge-Norvege

Apple-Velkommen_til_Snow_Leopard.pdf-Norge-Norvege.htm

Apple-TimeCapsule_Klargjoringsoversikt.pdf-Norge-Norvege

Apple-iPhone_3GS_Hurtigstart.pdf-Norge-Norvege

Apple-Snow_Leopard_Installeringsinstruksjoner.pdf-Norge-Norvege

Apple-iMacG5_iSight_UG.pdf-Norge-Norvege

Apple-iPod_Handbok_S0342141.pdf-Norge-Norvege

Apple-ipad_brukerhandbok.pdf-Norge-Norvege

Apple-GE_Money_Bank_Handlekonto.pdf-Norge-Norvege

Apple-MacBook_Air_11inch_Late2010_UG_H.pdf-Norge-Norvege

Apple-iPod_nano_6thgen_Brukerhandbok.pdf-Norge-Norvege

Apple-iPod_touch_iOS4_Brukerhandbok.pdf-Norge-Norvege

Apple-MacBook_Air_13inch_Late2010_UG_H.pdf-Norge-Norvege

Apple-MacBook_Pro_15inch_Early2011_H.pdf-Norge-Norvege

Apple-Numbers09_Brukerhandbok.pdf-Norge-Norvege

Apple-Welcome_to_Leopard.pdf-Japon

Apple-PowerMacG5_UserGuide.pdf-Norge-Norvege

Apple-iPod_touch_2.1_Brukerhandbok.pdf-Norge-Norvege

Apple-Boot_Camp_Installering-klargjoring.pdf-Norge-Norvege

Apple-MacOSX10.3_Welcome.pdf-Norge-Norvege

Apple-iPod_shuffle_3rdGen_UG_H.pdf-Norge-Norvege

Apple-iPhone_4_Viktig_produktinformasjon.pdf-Norge-Norvege

Apple_TV_Klargjoringsoversikt.pdf-Norge-Norvege

Apple-iMovie_08_Komme_i_gang.pdf-Norge-Norvege

Apple-iPod_classic_160GB_Brukerhandbok.pdf-Norge-Norvege

Apple-Boot_Camp_Installering_10.6.pdf-Norge-Norvege

Apple-Network-Services-Location-Manager-Veiledning-for-nettverksadministratorer-Norge-Norvege

Apple-iOS_Business_Mar12_FR.pdf

Apple-PCIDualAttachedFDDICard.pdf

Apple-Aperture_Installing_Your_Software_f.pdf

Apple-User_Management_Admin_v10.4.pdf

Apple-Compressor-4-ユーザーズマニュアル Japon

Apple-Network_Services_v10.4.pdf

Apple-iPod_2ndGen_USB_Power_Adapter-DE

Apple-Mail_Service_v10.4.pdf

Apple-AirPort_Express_Opstillingsvejledning_5.1.pdf

Apple-MagSafe_Airline_Adapter.pdf

Apple-L-Apple-Multiple-Scan-20-Display

Apple-Administration_du_service_de_messagerie_10.5.pdf

Apple-System_Image_Admin.pdf

Apple-iMac_Intel-based_Late2006.pdf-Japon

Apple-iPhone_3GS_Finger_Tips_J.pdf-Japon

Apple-Power-Mac-G4-Mirrored-Drive-Doors-Japon

Apple-AirMac-カード取り付け手順-Japon

Apple-iPhone開発ガイド-Japon

Apple-atadrive_pmg4mdd.j.pdf-Japon

Apple-iPod_touch_2.2_User_Guide_J.pdf-Japon

Apple-Mac_OS_X_Server_v10.2.pdf

Apple-AppleCare_Protection_Plan_for_Apple_TV.pdf

Apple_Component_AV_Cable.pdf

Apple-DVD_Studio_Pro_4_Installation_de_votre_logiciel

Apple-Windows_Services

Apple-Motion_3_New_Features_F

Apple-g4mdd-fw800-lowerfan

Apple-MacOSX10.3_Welcome

Apple-Print_Service

Apple-Xserve_Setup_Guide_F

Apple-PowerBookG4_17inch1.67GHzUG

Apple-iMac_Intel-based_Late2006

Apple-Installation_de_votre_logiciel

Apple-guide_des_fonctions_de_l_iPod_nano

Apple-Administration_de_serveur_v10.5

Apple-Mac-OS-X-Server-Premiers-contacts-Pour-la-version-10.3-ou-ulterieure

Apple-boot_camp_install-setup

Apple-iBookG3_14inchUserGuideMultilingual

Apple-mac_pro_server_mid2010_ug_f

Apple-Motion_Supplemental_Documentation

Apple-imac_mid2011_ug_f

Apple-iphone_guide_de_l_utilisateur

Apple-macbook_air_11inch_mid2011_ug_fr

Apple-NouvellesfonctionnalitesdeLogicExpress7.2

Apple-QT_Streaming_Server

Apple-Web_Technologies_Admin

Apple-Mac_Pro_Early2009_4707_UG

Apple-guide_de_l_utilisateur_de_Numbers08

Apple-Decouverte_d_Aperture_2

Apple-Guide_de_configuration_et_d'administration

Apple-mac_integration_basics_fr_106.

Apple-iPod_shuffle_4thgen_Guide_de_l_utilisateur

Apple-ARA_Japan

Apple-081811_APP_iPhone_Japanese_v5.4.pdf-Japan

Apple-Recycle_Contract120919.pdf-Japan

Apple-World_Travel_Adapter_Kit_UG

Apple-iPod_nano_6thgen_User_Guide

Apple-RemoteSupportJP

Apple-Mac_mini_Early2009_UG_F.pdf-Manuel-de-l-utilisateur

Apple-Compressor_3_Batch_Monitor_User_Manual_F.pdf-Manuel-de-l-utilisateur

Apple-Premiers__contacts_avec_iDVD_08

Apple-Mac_mini_Intel_User_Guide.pdf

Apple-Prise_en_charge_des_surfaces_de_controle_Logic_Express_8

Apple-mac_integration_basics_fr_107.pdf

Apple-Final-Cut-Pro-7-Niveau-1-Guide-de-preparation-a-l-examen

Apple-Logic9-examen-prep-fr.pdf-Logic-Pro-9-Niveau-1-Guide-de-preparation-a-l-examen

Apple-aperture_photography_fundamentals.pdf-Manuel-de-l-utilisateu

Apple-emac-memory.pdf-Manuel-de-l-utilisateur

Apple-Apple-Installation-et-configuration-de-votre-Power-Mac-G4

Apple-Guide_de_l_administrateur_d_Xsan_2.pdf

Apple-premiers_contacts_avec_imovie6.pdf

Apple-Tiger_Guide_Installation_et_de_configuration.pdf

Apple-Final-Cut-Pro-7-Level-One-Exam-Preparation-Guide-and-Practice-Exam

Apple-Open_Directory.pdf

Apple-Nike_+_iPod_User_guide

Apple-ard_admin_guide_2.2_fr.pdf

Apple-systemoverviewj.pdf-Japon

Apple-Xserve_TO_J070411.pdf-Japon

Apple-Mac_Pro_User_Guide.pdf

Apple-iMacG5_iSight_UG.pdf

Apple-premiers_contacts_avec_iwork_08.pdf

Apple-services_de_collaboration_2e_ed_10.4.pdf

Apple-iPhone_Bluetooth_Headset_Benutzerhandbuch.pdf

Apple-Guide_de_l_utilisateur_de_Keynote08.pdf

APPLE/Apple-Logic-Pro-9-Effectsrfr.pdf

Apple-Logic-Pro-9-Effectsrfr.pdf

Apple-iPod_shuffle_3rdGen_UG_F.pdf

Apple-iPod_classic_160Go_Guide_de_l_utilisateur.pdf

Apple-iBookG4GettingStarted.pdf

Apple-Administration_de_technologies_web_10.5.pdf

Apple-Compressor-4-User-Manual-fr

Apple-MainStage-User-Manual-fr.pdf

Apple-Logic_Pro_8.0_lbn_j.pdf

Apple-PowerBookG4_15inch1.67-1.5GHzUserGuide.pdf

Apple-MacBook_Pro_15inch_Mid2010_CH.pdf

Apple-LED_Cinema_Display_27-inch_UG.pdf

Apple-MacBook_Pro_15inch_Mid2009_RS.pdf

Apple-macbook_pro_13inch_early2011_f.pdf

Apple-iMac_Mid2010_UG_BR.pdf

Apple-iMac_Late2009_UG_J.pdf

Apple-iphone_user_guide-For-iOS-6-Software

Apple-iDVD5_Getting_Started.pdf

Apple-guide_des_fonctionnalites_de_l_ipod_touch.pdf

Apple_iPod_touch_User_Guide

Apple_macbook_pro_13inch_early2011_f

Apple_Guide_de_l_utilisateur_d_Utilitaire_RAID

Apple_Time_Capsule_Early2009_Setup_F

Apple_iphone_4s_finger_tips_guide_rs

Apple_iphone_upute_za_uporabu

Apple_ipad_user_guide_ta

Apple_iPod_touch_User_Guide

apple_earpods_user_guide

apple_iphone_gebruikershandleiding

apple_iphone_5_info

apple_iphone_brukerhandbok

apple_apple_tv_3rd_gen_setup_tw

apple_macbook_pro-retina-mid-2012-important_product_info_ch

apple_Macintosh-User-s-Guide-for-Macintosh-PowerBook-145

Apple_ipod_touch_user_guide_ta

Apple_TV_2nd_gen_Setup_Guide_h

Apple_ipod_touch_manual_del_usuario

Apple_iphone_4s_finger_tips_guide_tu

Apple_macbook_pro_retina_qs_th

Apple-Manuel_de_l'utilisateur_de_Final_Cut_Server

Apple-iMac_G5_de_lutilisateur

Apple-Cinema_Tools_4.0_User_Manual_F

Apple-Personal-LaserWriter300-User-s-Guide

Apple-QuickTake-100-User-s-Guide-for-Macintosh

Apple-User-s-Guide-Macintosh-LC-630-DOS-Compatible

Apple-iPhone_iOS3.1_User_Guide

Apple-iphone_4s_important_product_information_guide

Apple-iPod_shuffle_Features_Guide_F

Liste-documentation-apple

Apple-Premiers_contacts_avec_iMovie_08

Apple-macbook_pro-retina-mid-2012-important_product_info_br

Apple-macbook_pro-13-inch-mid-2012-important_product_info

Apple-macbook_air-11-inch_mid-2012-qs_br

Apple-Manuel_de_l_utilisateur_de_MainStage

Apple-Compressor_3_User_Manual_F

Apple-Color_1.0_User_Manual_F

Apple-guide_de_configuration_airport_express_4.2

Apple-TimeCapsule_SetupGuide

Apple-Instruments_et_effets_Logic_Express_8

Apple-Manuel_de_l_utilisateur_de_WaveBurner

Apple-Macmini_Guide_de_l'utilisateur

Apple-PowerMacG5_UserGuide

Disque dur, ATA parallèle Instructions de remplacement

Apple-final_cut_pro_x_logic_effects_ref_f

Apple-Leopard_Installationshandbok

Manuale Utente PowerBookG4

Apple-thunderbolt_display_getting_started_1e

Apple-Compressor-4-Benutzerhandbuch

Apple-macbook_air_11inch_mid2011_ug

Apple-macbook_air-mid-2012-important_product_info_j

Apple-iPod-nano-Guide-des-fonctionnalites

Apple-iPod-nano-Guide-des-fonctionnalites

Apple-iPod-nano-Guide-de-l-utilisateur-4eme-generation

Apple-iPod-nano-Guide-de-l-utilisateur-4eme-generation

Apple-Manuel_de_l_utilisateur_d_Utilitaire_de_reponse_d_impulsion

Apple-Aperture_2_Raccourcis_clavier

AppleTV_Setup-Guide

Apple-livetype_2_user_manual_f

Apple-imacG5_17inch_harddrive

Apple-macbook_air_guide_de_l_utilisateur

Apple-MacBook_Early_2008_Guide_de_l_utilisateur

Apple-Keynote-2-Guide-de-l-utilisateur

Apple-PowerBook-User-s-Guide-for-PowerBook-computers

Apple-Macintosh-Performa-User-s-Guide-5200CD-and-5300CD

Apple-Macintosh-Performa-User-s-Guide

Apple-Workgroup-Server-Guide

Apple-iPod-nano-Guide-des-fonctionnalites

Apple-iPad-User-Guide-For-iOS-5-1-Software

Apple-Boot-Camp-Guide-d-installation-et-de-configuration

Apple-iPod-nano-Guide-de-l-utilisateur-4eme-generation

Power Mac G5 Guide de l’utilisateur APPLE

Guide de l'utilisateur PAGE '08 APPLE

Guide de l'utilisateur KEYNOTE '09 APPLE

Guide de l'Utilisateur KEYNOTE '3 APPLE

Guide de l'Utilisateur UTILITAIRE RAID

Guide de l'Utilisateur Logic Studio

Power Mac G5 Guide de l’utilisateur APPLE

Guide de l'utilisateur PAGE '08 APPLE

Guide de l'utilisateur KEYNOTE '09 APPLE

Guide de l'Utilisateur KEYNOTE '3 APPLE

Guide de l'Utilisateur UTILITAIRE RAID

Guide de l'Utilisateur Logic Studio

Guide de l’utilisateur ipad Pour le logiciel iOS 5.1

PowerBook G4 Premiers Contacts APPLE

Guide de l'Utilisateur iphone pour le logiciel ios 5.1 APPLE

Guide de l’utilisateur ipad Pour le logiciel iOS 4,3

Guide de l’utilisateur iPod nano 5ème génération

Guide de l'utilisateur iPod Touch 2.2 APPLE

Guide de l’utilisateur QuickTime 7  Mac OS X 10.3.9 et ultérieur Windows XP et Windows 2000

Guide de l'utilisateur MacBook 13 pouces Mi 2010

Guide de l’utilisateur iPhone (Pour les logiciels iOS 4.2 et 4.3)

Guide-de-l-utilisateur-iPod-touch-pour-le-logiciel-ios-4-3-APPLE

Guide-de-l-utilisateur-iPad-2-pour-le-logiciel-ios-4-3-APPLE

Guide de déploiement en entreprise iPhone OS

Guide-de-l-administrateur-Apple-Remote-Desktop-3-1

Guide-de-l-utilisateur-Apple-Xserve-Diagnostics-Version-3X103

Guide-de-configuration-AirPort-Extreme-802.11n-5e-Generation

Guide-de-configuration-AirPort-Extreme-802-11n-5e-Generation

Guide-de-l-utilisateur-Capteur-Nike-iPod

Guide-de-l-utilisateur-iMac-21-5-pouces-et-27-pouces-mi-2011-APPLE

Guide-de-l-utilisateur-Apple-Qadministrator-4

Guide-d-installation-Apple-TV-3-eme-generation

User-Guide-iPad-For-ios-5-1-Software

Resource Programming GuideContents About Resources 5 At a Glance 5 Nib Files Store the Objects of Your Application’s User Interface 6 String Resources Containing Localizable Text 6 Images, Sounds, and Movies Represent Pre-rendered Content 6 Property Lists and Data Files Separate Data from Code 7 iOS Supports Device-Specific Resources 7 See Also 8 Nib Files 9 Anatomy of a Nib File 9 About Your Interface Objects 10 About the File’s Owner 10 About the First Responder 10 About the Top-Level Objects 11 About Image and Sound Resources 11 Nib File Design Guidelines 11 The Nib Object Life Cycle 12 The Object Loading Process 12 Managing the Lifetimes of Objects from Nib Files 15 Top-level Objects in OS X May Need Special Handling 17 Legacy Patterns 17 Action Methods 20 Built-In Support For Nib Files 21 The Application Loads the Main Nib File 22 Each View Controller Manages its Own Nib File 22 Document and Window Controllers Load Their Associated Nib File 22 Loading Nib Files Programmatically 23 Loading Nib Files Using NSBundle 23 Getting a Nib File’s Top-Level Objects 25 Loading Nib Files Using UINib and NSNib 27 Replacing Proxy Objects at Load Time 28 Accessing the Contents of a Nib File 30 Connecting Menu Items Across Nib Files 30 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 2String Resources 32 Creating Strings Resource Files 33 Choosing Which Strings to Localize 33 About the String-Loading Macros 34 Using the genstrings Tool to Create Strings Files 35 Creating Strings Files Manually 36 Detecting Non-localizable Strings 37 Loading String Resources Into Your Code 37 Using the Core Foundation Framework 38 Using the Foundation Framework 39 Examples of Getting Strings 39 Advanced Strings File Tips 40 Searching for Custom Functions With genstrings 40 Formatting String Resources 41 Using Special Characters in String Resources 41 Debugging Strings Files 42 Image, Sound, and Video Resources 43 Images and Sounds in Nib Files 43 Loading Image Resources 43 Loading Images in Objective-C 44 Loading Images Using Quartz 45 Specifying High-Resolution Images in iOS 46 Data Resource Files 48 Property List Files 48 OS X Data Resource Files 48 Document Revision History 50 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 3 ContentsTables and Listings Nib Files 9 Listing 1-1 Loading a nib file from the current bundle 24 Listing 1-2 Loading a nib in an iPhone application 25 Listing 1-3 Using outlets to get the top-level objects 25 Listing 1-4 Getting the top-level objects from a nib file at runtime 26 Listing 1-5 Loading a nib file using NSNib 28 Listing 1-6 Replacing placeholder objects in a nib file 29 String Resources 32 Table 2-1 Common parameters found in string-loading routines 37 Listing 2-1 A simple strings file 32 Listing 2-2 Strings localized for English 36 Listing 2-3 Strings localized for German 36 Listing 2-4 Strings with formatting characters 41 Image, Sound, and Video Resources 43 Listing 3-1 Loading an image resource 44 Listing 3-2 Using data providers to load image resources 45 Data Resource Files 48 Table 4-1 Other resource types 49 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 4Applied to computer programs, resources are data filesthat accompany a program’s executable code. Resources simplify the code you have to write by moving the creation of complex sets of data or graphical content outside of your code and into more appropriate tools. For example, rather than creating images pixel by pixel using code, it is much more efficient (and practical) to create them in an image editor. To take advantage of a resource, all your code has to do is load it at runtime and use it. In addition to simplifying your code, resources are also an intimate part of the internationalization process for all applications. Rather than hard-coding strings and other user-visible content in your application, you can place that content in external resource files. Localizing your application then becomes a simple process of creating new versions of each resource file for each supported language. The bundle mechanism used in both OS X and iOS provides a way to organize localized resources and to facilitate the loading of resource files that match the user’s preferred language. This document provides information about the types of resources supported in OS X and iOS and how you use those resources in your code. This document does not focus on the resource-creation process. Most resources are created using either third-party applications or the developer tools provided in the /Developer/Applications directory. In addition, although this document refers to the use of resources in applications, the information also applies to other types of bundled executables, including frameworks and plug-ins. Before reading this document, you should be familiar with the organizationalstructure imposed by application bundles. Understanding this structure makes it easier to organize and find the resource files your application uses. For information on the structure of bundles, see Bundle Programming Guide . At a Glance Applications can contain many types of resources but there are several that are supported directly by iOS and OS X. 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 5 About ResourcesNib Files Store the Objects of Your Application’s User Interface Nib files are the quintessential resource type used to create iOS and Mac apps. A nib file is a data archive that essentially contains a set of freeze-dried objects that you want to recreate at runtime. Nib files are used most commonly to store preconfigured windows, views, and other visually oriented objects but they can also store nonvisual objects such as controllers. You create nib files using the Interface Builder application, which provides a graphical assembly area for assembling your objects. When you subsequently load a nib file into your application, the nib-loading code instantiates each object in the file and restores it to the state you specified in Interface Builder. Thus, what you see in Interface Builder is really what you get in your application at runtime. Relevant Chapters: “Nib Files” (page 9) String Resources Containing Localizable Text Text is a prominent part of most user interfaces but also a resource that is most affected by localization changes. Rather than hard-coding text into your code, iOS and OS X support the storage of user-visible text in strings files, which are human-readable text files (in the UTF-16 encoding) containing a set of string resources for an application. (The use of the plural “strings” in is deliberate and due to the .strings filename extension used by files of that type.) Strings files greatly simplify the internationalization and localization process by allowing you to write your code once and then load the appropriately localized text from resource files that can be changed easily. The Core Foundation and Foundation frameworks provide the facilities for loading text from strings files. Applications that use these facilities can also take advantage of tools that come with Xcode to generate and maintain these resource files throughout the development process. Relevant Chapters: “String Resources” (page 32) Images, Sounds, and Movies Represent Pre-rendered Content Images, sound, and movie resources play an important role in iOS and Mac apps. Images are responsible for creating the unique visual style used by each operating system; they also help simplify your drawing code for complex visual elements. Sound and movie files similarly help enhance the overall user experience of your application while simplifying the code needed to create that experience. Both operating systems provide extensive support for loading and presenting these types of resources in your applications. About Resources At a Glance 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 6Relevant Chapters: “Image, Sound, and Video Resources” (page 43) Property Lists and Data Files Separate Data from Code A property list file is a structured file used to store string, number, Boolean, date, and raw data values. Data items in the file are organized using array and dictionary structures with most items associated with a unique key. The system uses property lists to store simple data sets. For example, the Info.plist file found in nearly every application is an example of a property list file. You can also use property list files for simple data storage needs. In addition to property lists, OS X supports some specially structured files for specific uses. For example, AppleScript data and user help are stored using specially formatted data files. You can also create custom data files of your own. Relevant Chapters: “Data Resource Files” (page 48) iOS Supports Device-Specific Resources In iOS 4.0 and later, it is possible to mark individual resource files as usable only on a specific type of device. This capability simplifies the code you have to write for Universal applications. Rather than creating separate code paths to load one version of a resource file for iPhone and a different version of the file for iPad, you can let the bundle-loading routines choose the correct file. All you have to do is name your resource files appropriately. To associate a resource file with a particular device, you add a custom modifier string to its filename. The inclusion of this modifier string yields filenames with the following format: . The string represents the original name of the resource file. It also represents the name you use when accessing the file from your code. Similarly, the string is the standard filename extension used to identify the type of the file. The string is a case-sensitive string that can be one of the following values: ● ~ipad - The resource should be loaded on iPad devices only. ● ~iphone - The resource should be loaded on iPhone or iPod touch devices only. About Resources At a Glance 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 7You can apply device modifiers to any type of resource file. For example, suppose you have an image named MyImage.png. To specify different versions of the image for iPad and iPhone, you would create resource files with the names MyImage~ipad.png and MyImage~iphone.png and include them both in your bundle. To load the image, you would continue to refer to the resource as MyImage.png in your code and let the system choose the appropriate version, as shown here: UIImage* anImage = [UIImage imageNamed:@"MyImage.png"]; On an iPhone or iPod touch device, the system loads the MyImage~iphone.png resource file, while on iPad, it loadsthe MyImage~ipad.png resource file. If a device-specific version of a resource is not found, the system falls back to looking for a resource with the original filename, which in the preceding example would be an image named MyImage.png. See Also The following Apple Developer documents are conceptually related to Resource Programming Guide : ● Bundle Programming Guide describes the bundle structure used by applications to store executable code and resources. ● Internationalization Programming Topics describes the process of preparing an application (and its resources) for translation into other languages. ● Interface Builder User Guide describes the application used to create nib file resources. ● Property List Programming Guide describes the facilities in place for loading property-list resource files into a Cocoa application. ● Property List Programming TopicsforCore Foundation describesthe facilitiesinplace forloadingproperty-list resource files into a C-based application. About Resources See Also 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 8Nib files play an important role in the creation of applications in OS X and iOS. With nib files, you create and manipulate your user interfaces graphically, using Xcode, instead of programmatically. Because you can see the results of your changesinstantly, you can experiment with different layouts and configurations very quickly. You can also change many aspects of your user interface later without rewriting any code. For applications built using the AppKit or UIKit frameworks, nib files take on an extra significance. Both of these frameworks support the use of nib files both for laying out windows, views, and controls and for integrating those items with the application’s event handling code. Xcode works in conjunction with these frameworks to help you connect the controls of your user interface to the objects in your project that respond to those controls. This integration significantly reduces the amount of setup that is required after a nib file is loaded and also makes it easy to change the relationships between your code and user interface later. Note: Although you can create an Objective-C application without using nib files, doing so is very rare and not recommended. Depending on your application, avoiding nib files might require you to replace large amounts of framework behavior to achieve the same results you would get using a nib file. Anatomy of a Nib File A nib file describes the visual elements of your application’s user interface, including windows, views, controls, and many others. It can also describe non-visual elements,such asthe objectsin your application that manage your windows and views. Most importantly, a nib file describes these objects exactly as they were configured in Xcode. At runtime, these descriptions are used to recreate the objects and their configuration inside your application. When you load a nib file at runtime, you get an exact replica of the objectsthat were in your Xcode document. The nib-loading code instantiates the objects, configures them, and reestablishes any inter-object connections that you created in your nib file. The following sections describe how nib files used with the AppKit and UIKit frameworks are organized, the types of objects found in them, and how you use those objects effectively. 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 9 Nib FilesAbout Your Interface Objects Interface objects are what you add to an nib file to implement your user interface. When a nib is loaded at runtime, the interface objects are the objects actually instantiated by the nib-loading code. Most new nib files have at least one interface object by default, typically a window or menu resource, and you add more interface objects to a nib file as part of your interface design. This is the most common type of object in a nib file and is typically why you create nib files in the first place. Besides representing visual objects, such as windows, views, controls, and menus, interface objects can also represent non-visual objects. In nearly all cases, the non-visual objects you add to a nib file are extra controller objects that your application uses to manage the visual objects. Although you could create these objects in your application, it is often more convenient to add them to a nib file and configure them there. Xcode provides a generic object that you use specifically when adding controllers and other non-visual objects to a nib file. It also provides the controller objects that are typically used to manage Cocoa bindings. About the File’s Owner One of the most important objects in a nib file is the File’s Owner object. Unlike interface objects, the File’s Owner object is a placeholder object that is not created when the nib file is loaded. Instead, you create this object in your code and pass it to the nib-loading code. The reason this object is so important is that it is the main link between your application code and the contents of the nib file. More specifically, it is the controller object that is responsible for the contents of the nib file. In Xcode, you can create connections between the File’s Owner and the other interface objects in your nib file. When you load the nib file, the nib-loading code recreates these connections using the replacement object you specify. This allows your object to reference objects in the nib file and receive messages from the interface objects automatically. About the First Responder In a nib file, the First Responder is a placeholder object that represents the first object in your application’s dynamically determined responder chain. Because the responder chain of an application cannot be determined at design time, the First Responder placeholder acts as a stand-in target for any action messages that need to be directed at the application’sresponder chain. Menu items commonly target the First Responder placeholder. For example, the Minimize menu item in the Window menu hides the frontmost window in an application, not just a specific window, and the Copy menu item should copy the current selection, not just the selection of a single control or view. Other objects in your application can target the First Responder as well. When you load a nib file into memory, there is nothing you have to do to manage or replace the First Responder placeholder object. The AppKit and UIKit frameworks automatically set and maintain the first responder based on the application’s current configuration. Nib Files Anatomy of a Nib File 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 10For more information about the responder chain and how it is used to dispatch events in AppKit–based applications,see “Event Architecture” inCocoa Event Handling Guide . For information about the responder chains and handling actions in iPhone applications, see Event Handling Guide for iOS . About the Top-Level Objects When your program loads a nib file, Cocoa recreates the entire graph of objects you created in Xcode. This object graph includes all of the windows, views, controls, cells, menus, and custom objects found in the nib file. The top-level objects are the subset of these objects that do not have a parent object. The top-level objects typically include only the windows, menubars, and custom controller objects that you add to the nib file. (Objects such as File’s Owner, First Responder, and Application are placeholder objects and not considered top-level objects.) Typically, you use outlets in the File’s Owner object to store references to the top-level objects of a nib file. If you do not use outlets, however, you can retrieve the top-level objects from the nib-loading routines directly. You should always keep a pointer to these objects somewhere because your application is responsible for releasing them when it is done using them. For more information about the nib object behavior at load time, see “Managing the Lifetimes of Objects from Nib Files” (page 15). About Image and Sound Resources In Xcode, you can refer to external image and sound resources from within the contents of your nib files. Some controls and views are able to display images or play sounds as part of their default configuration. The Xcode library provides access to the image and sound resources of your Xcode projects so that you can link your nib files to these resources. The nib file does not store these resources directly. Instead, it stores the name of the resource file so that the nib-loading code can find it later. When you load a nib file that contains references to image or sound resources, the nib-loading code reads the actual image or sound file into memory and and caches it. In OS X, image and sound resources are stored in named caches so that you can access them later if needed. In iOS, only image resources are stored in named caches. To access images, you use the imageNamed: method of NSImage or UIImage, depending on your platform. To access cached sounds in OS X, use the soundNamed: method of NSSound. Nib File Design Guidelines When creating your nib files, it is important to think carefully about how you intend to use the objects in that file. A very simple application might be able to store all of its user interface components in a single nib file, but for most applications, it is better to distribute components across multiple nib files. Creating smaller nib files lets you load only those portions of your interface that you need immediately. They also make it easier to debug any problems you might encounter, since there are fewer places to look for problems. Nib Files Nib File Design Guidelines 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 11When creating your nib files, try to keep the following guidelines in mind: ● Design your nib files with lazy loading in mind. Plan on loading nib files that contain only those objects you need right away. ● In the main nib file for an OS X application, considerstoring only the application menu bar and an optional application delegate object in the nib file. Avoid including any windows or user-interface elements that will not be used until after the application has launched. Instead, place those resources in separate nib files and load them as needed after launch. ● Store repeated user-interface components (such as document windows) in separate nib files. ● For a window or menu that is used only occasionally,store it in a separate nib file. By storing it in a separate nib file, you load the resource into memory only if it is actually used. ● Make the File’s Owner the single point-of-contact for anything outside of the nib file; see “Accessing the Contents of a Nib File” (page 30). The Nib Object Life Cycle When a nib file is loaded into memory, the nib-loading code takes several steps to ensure the objects in the nib file are created and initialized properly. Understanding these steps can help you write better controller code to manage your user interfaces. The Object Loading Process When you use the methods of NSNib or NSBundle to load and instantiate the objectsin a nib file, the underlying nib-loading code does the following: 1. It loads the contents of the nib file and any referenced resource files into memory: ● The raw data for the entire nib object graph is loaded into memory but is not unarchived. ● Any custom image resources associated with the nib file are loaded and added to the Cocoa image cache; see “About Image and Sound Resources” (page 11). ● Any custom sound resources associated with the nib file are loaded and added to the Cocoa sound cache; see “About Image and Sound Resources” (page 11). 2. It unarchives the nib object graph data and instantiates the objects. How it initializes each new object depends on the type of the object and how it was encoded in the archive. The nib-loading code uses the following rules (in order) to determine which initialization method to use. a. By default, objects receive an initWithCoder: message. Nib Files The Nib Object Life Cycle 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 12In OS X, the list of standard objects includes the views, cells, menus, and view controllers that are provided by the system and available in the default Xcode library. It also includes any third-party objects that were added to the library using a custom plug-in. Even if you change the class of such an object, Xcode encodes the standard object into the nib file and then tells the archiver to swap in your custom class when the object is unarchived. In iOS, any object that conforms to the NSCoding protocol is initialized using the initWithCoder: method. This includes all subclasses of UIView and UIViewController whether they are part of the default Xcode library or custom classes you define. b. Custom views in OS X receive an initWithFrame: message. Custom views are subclasses of NSView for which Xcode does not have an available implementation. Typically, these are viewsthat you define in your application and use to provide custom visual content. Custom views do not include standard system views(like NSSlider) that are part of the default library or part of an integrated third-party plug-in. When it encounters a custom view, Xcode encodes a special NSCustomView object into your nib file. The custom view object includesthe information it needsto build the real view subclass you specified. At load time, the NSCustomView object sends an alloc and initWithFrame: message to the real view class and then swaps the resulting view object in for itself. The net effect is that the real view object handles subsequent interactions during the nib-loading process. Custom views in iOS do not use the initWithFrame: method for initialization. c. Custom objects other than those described in the preceding steps receive an init message. 3. It reestablishes all connections(actions, outlets, and bindings) between objectsin the nib file. Thisincludes connections to File’s Owner and other placeholder objects. The approach for establishing connections differs depending on the platform: ● Outlet connections ● In OS X, the nib-loading code tries to reconnect outlets using the object’s own methods first. For each outlet, Cocoa looks for a method of the form setOutletName: and calls it if such a method is present. If it cannot find such a method, Cocoa searchesthe object for an instance variable with the corresponding outlet name and tries to set the value directly. If the instance variable cannot be found, no connection is created. In OS X v10.5 and later, setting an outlet also generates a key-value observing (KVO) notification for any registered observers. These notifications may occur before all inter-object connections are reestablished and definitely occur before any awakeFromNib methods of the objects have been called. Prior to v10.5, these notifications are not generated. For more information about KVO notifications, see Key-Value Observing Programming Guide . Nib Files The Nib Object Life Cycle 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 13● In iOS, the nib-loading code uses the setValue:forKey: method to reconnect each outlet. That method similarly looks for an appropriate accessor method and falls back on other means when that fails. For more information about how this method sets values, see its description in NSKeyValueCoding Protocol Reference . Setting an outlet in iOS also generates a KVO notification for any registered observers. These notifications may occur before all inter-object connections are reestablished and definitely occur before any awakeFromNib methods of the objects have been called. For more information about KVO notifications, see Key-Value Observing Programming Guide . ● Action connections ● In OS X, the nib-loading code uses the source object’s setTarget: and setAction: methods to establish the connection to the target object. If the target object does not respond to the action method, no connection is created. If the target object is nil, the action is handled by the responder chain. ● In iOS, the nib-loading code uses the addTarget:action:forControlEvents: method of the UIControl object to configure the action. If the target is nil, the action is handled by the responder chain. ● Bindings ● In OS X, Cocoa usesthe bind:toObject:withKeyPath:options: method of the source object to create the connection between it and its target object. ● Bindings are not supported in iOS. 4. It sends an awakeFromNib message to the appropriate objects in the nib file that define the matching selector: ● In OS X, this message is sent to any interface objects that define the method. It is also sent to the File’s Owner and any placeholder objects that define it as well. ● In iOS, this message is sent only to the interface objects that were instantiated by the nib-loading code. It is not sent to File’s Owner, First Responder, or any other placeholder objects. 5. It displays any windows whose “Visible at launch time” attribute was enabled in the nib file. The order in which the nib-loading code calls the awakeFromNib methods of objects is not guaranteed. In OS X, Cocoa tries to call the awakeFromNib method of File’s Owner last but does not guarantee that behavior. If you need to configure the objects in your nib file further at load time, the most appropriate time to do so is after your nib-loading call returns. At that point, all of the objects are created, initialized, and ready for use. Nib Files The Nib Object Life Cycle 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 14Managing the Lifetimes of Objects from Nib Files Each time you ask the NSBundle or NSNib class to load a nib file, the underlying code creates a new copy of the objects in that file and returns them to you. (The nib-loading code does not recycle nib file objects from a previous load attempt.) You need to ensure that you maintain the new object graph as long as necessary, and disown it when you are finished with it. You typically need strong references to top-level objects to ensure that they are not deallocated; you don’t need strong references to objects lower down in the graph because they’re owned by their parents, and you should minimize the risk of creating strong reference cycles. From a practical perspective, in iOS and OS X outlets should be defined as declared properties. Outlets should generally be weak, except for those from File’s Owner to top-level objects in a nib file (or, in iOS, a storyboard scene) which should be strong. Outlets that you create should therefore typically be weak, because: ● Outlets that you create to subviews of a view controller’s view or a window controller’s window, for example, are arbitrary references between objects that do not imply ownership. ● The strong outlets are frequently specified by framework classes (for example, UIViewController’s view outlet, or NSWindowController’s window outlet). @property (weak) IBOutlet MyView *viewContainerSubview; @property (strong) IBOutlet MyOtherClass *topLevelObject; Note: InOS X, not all classessupport weak references; these are NSATSTypesetter, NSColorSpace, NSFont, NSFontManager, NSFontPanel, NSImage, NSMenuView, NSParagraphStyle, NSSimpleHorizontalTypesetter, NSTableCellView, NSTextView, NSViewController, NSWindow, and NSWindowController, and all classes in the AV Foundation framework. In cases where you cannot therefore specify weak, you should instead use assign: @property (assign) IBOutlet NSTextView *textView; Outlets may be considered private to the defining class; if you prefer, you can hide the property declarations a class extension. For example: // MyClass.h @interface MyClass : MySuperclass @end Nib Files Managing the Lifetimes of Objects from Nib Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 15// MyClass.m @interface MyClass () @property (weak) IBOutlet MyView *viewContainerSubview; @property (strong) IBOutlet MyOtherClass *topLevelObject; @end These patterns extend to references from a container view to its subviews where you have to consider the internal consistency of your object graph. For example, in the case of a table view cell, outlets to specific subviews should again typically be weak. If a table view contains an image view and a text view, then these remain valid so long as they are subviews of the table view cell itself. Outlets should be changed to strong when the outlet should be considered to own the referenced object: ● Asindicated previously, thisis often the case with File’s Owner—top level objectsin a nib file are frequently considered to be owned by the File’s Owner. ● You may in some situations need an object from a nib file to exist outside of its original container. For example, you might have an outlet for a view that can be temporarily removed from itsinitial view hierarchy and must therefore be maintained independently. Classes that you expect to be subclassed (in particular abstract classes) expose outlets publicly so that they can be used appropriately by subclasses (e.g. UIViewController’s view outlet). Outlets might also be exposed where there is an expectation that consumers of the class will need to interact with the property; for example a table view cell might expose subviews. In thislatter case, it may be appropriate to expose a read-only public outlet that is redefined privately as read-write, for example: // MyClass.h @interface MyClass : UITableViewCell @property (weak, readonly) MyType *outletName; @end // MyClass.m @interface MyClass () @property (weak, readwrite) IBOutlet MyType *outletName; @end Nib Files Managing the Lifetimes of Objects from Nib Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 16Top-level Objects in OS X May Need Special Handling For historical reasons, in OS X the top-level objects in a nib file are created with an additional reference count. The Application Kit offers a couple of features that help to ensure that nib objects are properly released: ● NSWindow objects (including panels) have an isReleasedWhenClosed attribute, which if set to YES instructs the window to release itself (and consequently all dependent objects in its view hierarchy) when it is closed. In the nib file, you set this option through the “Release when closed” check box in the Attributes pane of the Xcode inspector. ● If the File’s Owner of a nib file is an NSWindowController object (the default in document nibs in document-based applications—recall that NSDocument manages an instance of NSWindowController) or an NSViewController object, it automatically disposes of the windows it manages. If the File’s Owner is not an instance of NSWindowController or NSViewController, then you need to decrement the reference count of the top level objects yourself. With manual reference counting, it was possible to achieve this by sending top-level objects a release message. You cannot do this with ARC. Instead, you cast references to top-level objects to a Core Foundation type and use CFRelease. (If you don’t want to have outlets to all top-level objects, you can use the instantiateNibWithOwner:topLevelObjects: method of the NSNib class to get an array of a nib file’s top-level objects.) Legacy Patterns Prior to ARC, the rules for managing nib objects are different from those described above. How you manage the objects depends on the platform and on the memory model in use. Whichever platform you develop for, you should define outlets using the Objective-C declared properties feature. The general form of the declaration should be: @property (attributes) IBOutlet UserInterfaceElementClass *anOutlet; Because the behavior of outlets depends on the platform, the actual declaration differs: ● For iOS, you should use: @property (nonatomic, retain) IBOutlet UserInterfaceElementClass *anOutlet; ● For OS X, you should use: @property (assign) IBOutlet UserInterfaceElementClass *anOutlet; Nib Files Managing the Lifetimes of Objects from Nib Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 17You should then either synthesize the corresponding accessor methods, or implement them according to the declaration, and (in iOS) release the corresponding variable in dealloc. This pattern also works if you use the modern runtime and synthesize the instance variables, so it remains consistent across all situations. Managing Nib Objects in iOS Top-Level Objects Objects in the nib file are created with a retain count of 1 and then autoreleased. As it rebuilds the object hierarchy, UIKit reestablishes connections between the objects using setValue:forKey:, which uses the available setter method or retains the object by default if no setter method is available. This means that (assuming you follow the pattern shown above) any object for which you have an outlet remains valid. If there are any top-level objects you do not store in outlets, however, you must retain either the array returned by the loadNibNamed:owner:options: method or the objects inside the array to prevent those objects from being released prematurely. Memory Warnings When a view controller receives a memory warning (didReceiveMemoryWarning), it should relinquish ownership of resources that are currently not needed and that can be recreated later if required. One such resource is the view controller's view itself. If it does not have a superview, the view is disposed of (in its implementation of didReceiveMemoryWarning, UIViewController invokes [self setView:nil]). Because outlets to elements within the nib file are typically retained, however, even though the main view is disposed of, absent any further action the outlets are not disposed of. This is not in and of itself a problem—if and when the main view is reloaded, they will simply be replaced—but it does mean that the beneficial effect of the didReceiveMemoryWarning is reduced. To ensure that you properly relinquish ownership of outlets, in your custom view controller class you can implement viewDidUnload to invoke your accessor methods to set outlets to nil. - (void)viewDidUnload { self.anOutlet = nil; [super viewDidUnload]; } Nib Files Managing the Lifetimes of Objects from Nib Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 18Managing Nib Objects in OS X In OS X, the File’s Owner of a nib file is by default responsible for releasing the top-level objects in a nib file as well as any non-object resources created by the objects in the nib. The release of the root object of an object graph sets in motion the release of all dependent objects. The File’s Owner of an application’s main nib file (which containsthe application menu and possibly other items) isthe global application object NSApp. However, when a Cocoa application terminates, top level objects in the main nib do not automatically get dealloc messages just because NSApp is being deallocated. In other words, even in the main nib file, you have to manage the memory of top-level objects. The Application Kit offers a couple of features that help to ensure that nib objects are properly released: ● NSWindow objects (including panels) have an isReleasedWhenClosed attribute, which if set to YES instructs the window to release itself (and consequently all dependent objects in its view hierarchy) when it is closed. In the nib file, you set this option through the “Release when closed” check box in the Attributes pane of the inspector. ● If the File’s Owner of a nib file is an NSWindowController object (the default in document nibs in document-based applications—recall that NSDocument manages an instance of NSWindowController), it automatically disposes of the windows it manages. So in general, you are responsible for releasing top-level objects in a nib file. But in practice, if your nib file’s owner is an instance of NSWindowController it releases the top-level object for you. If one of your objects loads the nib itself (and the owner is not an instance of NSWindowController), you can define outlets to each top-level object so that at the appropriate time you can release them using those references. If you don’t wantto have outletsto alltop-level objects, you can use the instantiateNibWithOwner:topLevelObjects: method of the NSNib class to get an array of a nib file’s top-level objects. The issue of responsibility for nib object disposal becomes clearer when you consider the various kinds of applications. Most Cocoa applications are of two kinds: single window applications and document-based applications. In both cases, memory management of nib objects is automatically handled for you to some degree. With single-window applications, objects in the main nib file persist through the runtime life of the application and are released when the application terminates; however, dealloc is not guaranteed to be automatically invoked on objects from the main nib file when an application terminates. In document-based applications each document window is managed by an NSWindowController object which handles memory management for a document nib file. Some applications may have a more complex arrangement of nib files and top-level objects. For example, an application could have multiple nib files with multiple window controllers, loadable panels, and inspectors. But in most of these cases, if you use NSWindowController objects to manage windows and panels or if you set the “released when closed” window attribute, memory management is largely taken care of. If you decide against using window controllers and do not want to set the “release when closed” attribute, you should Nib Files Managing the Lifetimes of Objects from Nib Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 19explicitly free your nib file’s windows and other top-level objects when the window is closed. Also, if your application uses an inspector panel, (after being lazily loaded) the panel should typically persist throughout the lifetime of the application—there is no need to dispose of the inspector and its resources. Action Methods Broadly speaking, action methods(see Target-Action in OS X or Target-Action in iOS) are methodsthat are typically invoked by another object in a nib file. Action methods use type qualifier IBAction, which is used in place of the void return type, to flag the declared method as an action so that Xcode is aware of it. @interface MyClass - (IBAction)myActionMethod:(id)sender; @end You may choose to regard action methods as being private to your class and thus not declare them in the public @interface. (Because Xcode parses implementation files, there is no need to declare them in the header.) // MyClass.h @interface MyClass @end // MyClass.m @implementation MyClass - (IBAction)myActionMethod:(id)sender { // Implementation. } @end You should typically not invoke an action method programmatically. If your class needs to perform the work associated with the action method, then you should factor the implementation into a different method that is then invoked by the action method. // MyClass.h Nib Files Action Methods 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 20@interface MyClass @end // MyClass.m @interface MyClass (PrivateMethods) - (void)doSomething; - (void)doWorkThatRequiresMeToDoSomething; @end @implementation MyClass - (IBAction)myActionMethod:(id)sender { [self doSomething]; } - (void)doSomething { // Implementation. } - (void)doWorkThatRequiresMeToDoSomething { // Pre-processing. [self doSomething]; // Post-processing. } @end Built-In Support For Nib Files The AppKit and UIKit frameworks both provide a certain amount of automated behavior for loading and managing nib files in an application. Both frameworks provide infrastructure for loading an application’s main nib file. In addition, the AppKit framework providessupport for loading other nib filesthrough the NSDocument and NSWindowController classes. The following sections describe the built-in support for nib files, how you can take advantage of it, and ways to modify that support in your own applications. Nib Files Built-In Support For Nib Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 21The Application Loads the Main Nib File Most of the Xcode project templates for applications come preconfigured with a main nib file already in place. All you have to do is modify this default nib file in the nib file and build your application. At launch time, the application’s default configuration data tells the application object where to find this nib file so that it can load it. In applications based on either AppKit and UIKit, this configuration data is located in the application’s Info.plist file. When an application is first loaded, the default application startup code looks in the Info.plist file for the NSMainNibFile key. If it findsit, it looksin the application bundle for a nib file whose name (with or without the filename extension) matches the value of that key and loads it. Each View Controller Manages its Own Nib File The UIViewController (iOS) and NSViewController (OS X) classessupport the automatic loading of their associated nib file. If you specify a nib file when creating the view controller, that nib file isloaded automatically when you try to access the view controller’s view. Any connections between the view controller and the nib file objects are created automatically, and in iOS, the UIViewController object also receives additional notifications when the views are finally loaded and displayed on screen. To help manage memory better, the UIViewController class also handles the unloading of its nib file (as appropriate) during low-memory conditions. For more information about how you use the UIViewController class and how you configure it, see View Controller Programming Guide for iOS . Document and Window Controllers Load Their Associated Nib File In the AppKit framework, the NSDocument class works with the default window controller to load the nib file containing your document window. The windowNibName method of NSDocument is a convenience method that you can use to specify the nib file containing the corresponding document window. When a new document is created, the document object passes the nib file name you specify to the default window controller object, which loads and manages the contents of the nib file. If you use the standard templates provided by Xcode, the only thing you have to do is add the contents of your document window to the nib file. The NSWindowController class also provides automatic support for loading nib files. If you create custom window controllers programmatically, you have the option of initializing them with an NSWindow object or with the name of a nib file. If you choose the latter option, the NSWindowController class automatically loads the specified nib file the first time a client tries to access the window. After that, the window controller keeps the window around in memory; it does not reload it from the nib file, even if the window’s “Release when closed” attribute is set. Nib Files Built-In Support For Nib Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 22Important: When using either NSWindowController or NSDocument to load windows automatically, it is important that your nib file be configured correctly. Both classes include a window outlet that you must connect to the window you want them to manage. If you do not connect this outlet to a window object, the nib file is loaded but the document or window controller does not display the window. For more information about the Cocoa document architecture, see Document-Based App Programming Guide for Mac . Loading Nib Files Programmatically Both OS X and iOS provide convenience methods for loading nib files into your application. Both the AppKit and UIKit framework define additional methods on the NSBundle class that support the loading of nib files. In addition, the AppKit framework also provides the NSNib class, which provides similar nib-loading behavior as NSBundle but offers some additional advantages that might be useful in specific situations. As you plan out your application, make sure any nib files you plan to load manually are configured in a way that simplifies the loading process. Choosing an appropriate object for File’s Owner and keeping your nib files small can greatly improve their ease of use and memory efficiency. For more tips on configuring your nib files, see “Nib File Design Guidelines” (page 11). Loading Nib Files Using NSBundle The AppKit and UIKit frameworks define additional methods on the NSBundle class (using Objective-C categories) to support the loading of nib file resources. The semantics for how you use the methods differs between the two platforms as doesthe syntax for the methods. In AppKit, there are more optionsfor accessing bundles in general and so there are correspondingly more methods for loading nib files from those bundles. In UIKit, applications can load nib files only from their main bundle and so fewer options are needed. The methods available on the two platforms are as follows: ● AppKit ● loadNibNamed:owner: class method ● loadNibFile:externalNameTable:withZone: class method ● loadNibFile:externalNameTable:withZone: instance method ● UIKit ● loadNibNamed:owner:options: instance method Nib Files Loading Nib Files Programmatically 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 23Whenever loading a nib file, you should always specify an object to act as File’s Owner of that nib file. The role of the File’s Owner is an important one. It is the primary interface between your running code and the new objects that are about to be created in memory. All of the nib-loading methods provide a way to specify the File’s Owner, either directly or as a parameter in an options dictionary. One of the semantic differences between the way the AppKit and UIKit frameworks handle nib loading is the way the top-level nib objects are returned to your application. In the AppKit framework, you must explicitly request them using one of the loadNibFile:externalNameTable:withZone: methods. In UIKit, the loadNibNamed:owner:options: method returns an array of these objects directly. The simplest way to avoid having to worry about the top-level objects in either case is to store them in outlets of your File’s Owner object and to make sure the setter methods for those outlets retain their values. Because each platform uses different retain semantics, however, you must be sure to send the proper retain or release messages when appropriate. For information about the retention semantics for nib objects, see “Managing the Lifetimes of Objects from Nib Files” (page 15). Listing 1-1 shows a simple example of how to load a nib file using the NSBundle class in an AppKit–based application. As soon as the loadNibNamed:owner: method returns, you can begin using any outlets that refer to the nib file objects. In other words, the entire nib-loading process occurs within the confines of that single call. The nib-loading methods in the AppKit framework return a Boolean value to indicate whether the load operation was successful. Listing 1-1 Loading a nib file from the current bundle - (BOOL)loadMyNibFile { // The myNib file must be in the bundle that defines self's class. if (![NSBundle loadNibNamed:@"myNib" owner:self]) { NSLog(@"Warning! Could not load myNib file.\n"); return NO; } return YES; } Listing 1-2 shows an example of how to load a nib file in a UIKit–based application. In this case, the method checks the returned array to see if the nib objects were loaded successfully. (Every nib file should have at least one top-level object representing the contents of the nib file.) This example shows the simple case when the nib file contains no placeholder objects other than the File’s Owner object. For an example of how to specify additional placeholder objects, see “Replacing Proxy Objects at Load Time” (page 28). Nib Files Loading Nib Files Programmatically 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 24Listing 1-2 Loading a nib in an iPhone application - (BOOL)loadMyNibFile { NSArray* topLevelObjs = nil; topLevelObjs = [[NSBundle mainBundle] loadNibNamed:@"myNib" owner:self options:nil]; if (topLevelObjs == nil) { NSLog(@"Error! Could not load myNib file.\n"); return NO; } return YES; } Note: If you are developing a Universal application for iOS, you can use the device-specific naming conventionsto load the correct nib file for the underlying device automatically. For more information about how to name your nib files, see “iOS Supports Device-Specific Resources” (page 7). Getting a Nib File’s Top-Level Objects The easiest way to get the top-level objects of your nib file is to define outlets in the File’s Owner object along with setter methods (or better yet, properties) for accessing those objects. This approach ensures that the top-level objects are retained by your object and that you always have references to them. Listing 1-3 shows the interface and implementation of a simplified Cocoa class that uses an outlet to retain the nib file’s only top-level object. In this case, the only top-level object in the nib file is an NSWindow object. Because top-level objects in Cocoa have an initial retain count of 1, an extra release message is included. This is fine because by the time the release call is made, the property has already been retained the window. You would not want to release top-level objects in this manner in an iPhone application. Listing 1-3 Using outlets to get the top-level objects // Class interface @interface MyController : NSObject { NSWindow *window; } Nib Files Loading Nib Files Programmatically 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 25@property(retain) IBOutlet NSWindow *window; - (void)loadMyWindow; @end // Class implementation @implementation MyController // The synthesized property retains the window automatically. @synthesize window; - (void)loadMyWindow { [NSBundle loadNibNamed:@"myNib" owner:self]; // The window starts off with a retain count of 1 // and is then retained by the property, so add an extra release. [window release]; } @end If you do not want to use outlets to store references to your nib file’s top-level objects, you must retrieve those objects manually in your code. The technique for obtaining the top-level objects differs depending on the target platform. In OS X, you must ask for the objects explicitly, whereas in iOS they are returned to you automatically. Listing 1-4 showsthe processfor getting the top-level objects of a nib file in OS X. This method places a mutable array into the nameTable dictionary and associatesit with the NSNibTopLevelObjects key. The nib-loading code looks for this array object and, if present, places the top-level objects in it. Because each object starts with a retain count of 1 before it is added to the array, simply releasing the array is not enough to release the objects in the array as well. As a result, this method sends a release message to each of the objects to ensure that the array is the only entity holding a reference to them. Listing 1-4 Getting the top-level objects from a nib file at runtime - (NSArray*)loadMyNibFile { Nib Files Loading Nib Files Programmatically 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 26NSBundle* aBundle = [NSBundle mainBundle]; NSMutableArray* topLevelObjs = [NSMutableArray array]; NSDictionary* nameTable = [NSDictionary dictionaryWithObjectsAndKeys: self, NSNibOwner, topLevelObjs, NSNibTopLevelObjects, nil]; if (![aBundle loadNibFile:@"myNib" externalNameTable:nameTable withZone:nil]) { NSLog(@"Warning! Could not load myNib file.\n"); return nil; } // Release the objects so that they are just owned by the array. [topLevelObjs makeObjectsPerformSelector:@selector(release)]; return topLevelObjs; } Obtaining the top-level objects in an iPhone application is much simpler and is shown in Listing 1-2 (page 25). In the UIKit framework, the loadNibNamed:owner:options: method of NSBundle automatically returns an array with the top-level objects. In addition, by the time the array is returned, the retain counts on the objects are adjusted so that you do not need to send each object an extra release message. The returned array is the only owner of the objects. Loading Nib Files Using UINib and NSNib The UINib (iOS) and NSNib (OS X) classes provide better performance in situations where you want to create multiple copies of a nib file’s contents. The normal nib-loading process involves reading the nib file from disk and then instantiating the objects it contains. However, with the UINib and NSNib classes, the nib file is read from disk once and the contents are stored in memory. Because they are in memory, creating successive sets of objects takes less time because it does not require accessing the disk. Using the UINib and NSNib classes is always a two-step process. First, you create an instance of the class and initialize it with the nib file’s location information. Second, you instantiate the contents of the nib file to load the objects into memory. Each time you instantiate the nib file, you specify a different File’s Owner object and receive a new set of top-level objects. Nib Files Loading Nib Files Programmatically 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 27Listing 1-5 shows one way to load the contents of a nib file using the NSNib class in OS X. The array returned to you by the instantiateNibWithOwner:topLevelObjects: method comes already autoreleased. If you intend to use that array for any period of time, you should make a copy of it. Listing 1-5 Loading a nib file using NSNib - (NSArray*)loadMyNibFile { NSNib* aNib = [[NSNib alloc] initWithNibNamed:@"MyPanel" bundle:nil]; NSArray* topLevelObjs = nil; if (![aNib instantiateNibWithOwner:self topLevelObjects:&topLevelObjs]) { NSLog(@"Warning! Could not load nib file.\n"); return nil; } // Release the raw nib data. [aNib release]; // Release the top-level objects so that they are just owned by the array. [topLevelObjs makeObjectsPerformSelector:@selector(release)]; // Do not autorelease topLevelObjs. return topLevelObjs; } Replacing Proxy Objects at Load Time In iOS, it is possible to create nib files that include placeholder objects besides the File’s Owner. Proxy objects represent objects created outside of the nib file but which have some connection to the nib file’s contents. Proxies are commonly used to support navigation controllers in iPhone applications. When working with navigation controllers, you typically connect the File’s Owner object to some common object such as your application delegate. Proxy objects therefore represent the parts of the navigation controller object hierarchy that are already loaded in memory, because they were created programmatically or loaded from a different nib file. Nib Files Loading Nib Files Programmatically 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 28Note: Custom placeholder objects (other than File’s Owner) are not supported in OS X nib files. Each placeholder object you add to a nib file must have a unique name. To assign a name to an object, select the object in Xcode and open the inspector window. The Attributes pane of the inspector contains a Name field, which you use to specify the name for your placeholder object. The name you assign should be descriptive of the object’s behavior or type, but really it can be anything you want. When you are ready to load a nib file containing placeholder objects, you mustspecify the replacement objects for any proxies when you call the loadNibNamed:owner:options: method. The options parameter of this method accepts a dictionary of additional information. You use this dictionary to pass in the information about your placeholder objects. The dictionary must contain the UINibExternalObjects key whose value is another dictionary containing the name and object for each placeholder replacement. Listing 1-6 shows a sample version of an applicationDidFinishLaunching: method that loads the application’s main nib file manually. Because the application’s delegate object is created by the UIApplicationMain function, this method uses a placeholder (with the name “AppDelegate”) in the main nib file to represent that object. The proxies dictionary stores the placeholder object information and the options dictionary wraps that dictionary. Listing 1-6 Replacing placeholder objects in a nib file - (void)applicationDidFinishLaunching:(UIApplication *)application { NSArray* topLevelObjs = nil; NSDictionary* proxies = [NSDictionary dictionaryWithObject:self forKey:@"AppDelegate"]; NSDictionary* options = [NSDictionary dictionaryWithObject:proxies forKey:UINibExternalObjects]; topLevelObjs = [[NSBundle mainBundle] loadNibNamed:@"Main" owner:self options:options]; if ([topLevelObjs count] == 0) { NSLog(@"Warning! Could not load myNib file.\n"); return; } // Show window [window makeKeyAndVisible]; Nib Files Loading Nib Files Programmatically 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 29} For more information about the options dictionary of the loadNibNamed:owner:options: method, see NSBundle UIKit Additions Reference . Accessing the Contents of a Nib File Upon the successful loading a nib file, its contents become ready for you to use immediately. If you configured outlets in your File’s Owner to point to nib file objects, you can now use those outlets. If you did not configure your File’s Owner with any outlets, you should make sure you obtain a reference to the top-level objects in some manner so that you can release them later. Because outlets are populated with real objects when a nib file is loaded, you can subsequently use outlets as you would any other object you created programmatically. For example, if you have an outlet pointing to a window, you could send that window a makeKeyAndOrderFront: message to show it on the user’s screen. When you are done using the objects in your nib file, you must release them like any other objects. Important: You are responsible for releasing the top-level objects of any nib files you load when you are finished with those objects. Failure to do so is a cause of memory leaksin many applications. After releasing the top-level objects, it is a good idea to clear any outlets pointing to objects in the nib file by setting them to nil. You should clear outlets associated with all of the nib file’s objects, not just the top-level objects. Connecting Menu Items Across Nib Files The items in an OS X application’s menu bar often need to interact with many different objects, including your application’s documents and windows. The problem is that many of these objects cannot (or should not) be accessed directly from the main nib file. The File’s Owner of the main nib file is always set to an instance of the NSApplication class. And although you might be able to instantiate a number of custom objects in your main nib file, doing so is hardly practical or necessary. In the case of document objects, connecting directly to a specific document object is not even possible because the number of document objects can change dynamically and can even be zero. Most menu items send action messages to one of the following: ● A fixed object that always handles the command ● A dynamic object, such as a document or window Nib Files Connecting Menu Items Across Nib Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 30Messaging fixed objects is a relatively straightforward process that is usually best handled through the application delegate. The application delegate object assiststhe NSApplication objectin running the application and is one of the few objects that rightfully belongs in the main nib file. If the menu item refers to an application-level command, you can implement that command directly in the application delegate or just have the delegate forward the message to the appropriate object elsewhere in your application. If you have a menu item that acts on the contents of the frontmost window, you need to link the menu item to the First Responder placeholder object. If the action method associated with the menu item is specific to one of your objects(and not defined by Cocoa), you must add that action to the First Responder before creating the connection. After creating the connection, you need to implement the action method in your custom class. That object should also implement the validateMenuItem: method to enable the menu item at appropriate times. For more information about how the responder chain handles commands, see Cocoa Event Handling Guide . Nib Files Connecting Menu Items Across Nib Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 31An important part of the localization process is to localize all of the text strings displayed by your application. By their nature, strings located in nib files can be readily localized along with the rest of the nib file contents. Strings embedded in your code, however, must be extracted, localized, and then reinserted back into your code. To simplify this process—and to make the maintenance of your code easier—OS X and iOS provide the infrastructure needed to separate strings from your code and place them into resource files where they can be localized easily. Resource filesthat contain localizable strings are referred to as strings files because of their filename extension, which is .strings. You can create strings files manually or programmatically depending on your needs. The standard strings file format consists of one or more key-value pairs along with optional comments. The key and value in a given pair are strings of text enclosed in double quotation marks and separated by an equal sign. (You can also use a property list format for strings files. In such a case, the top-level node is a dictionary and each key-value pair of that dictionary is a string entry.) Listing 2-1 shows a simple strings file that contains non-localized entries for the default language. When you need to display a string, you pass the string on the left to one of the available string-loading routines. What you get back isthe matching value string containing the text translation that is most appropriate for the current user. For the development language, it is common to use the same string for both the key and value, but doing so is not required. Listing 2-1 A simple strings file /* Insert Element menu item */ "Insert Element" = "Insert Element"; /* Error string used for unknown error types. */ "ErrorString_1" = "An unknown error occurred."; A typical application has at least one strings file per localization, that is, one strings file in each of the bundle’s .lproj subdirectories. The name of the default strings file is Localizable.strings but you can create stringsfiles with any file name you choose. Creating stringsfilesis discussed in more depth in “Creating Strings Resource Files” (page 33). 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 32 String ResourcesNote: It is recommended that you save strings files using the UTF-16 encoding, which is the default encoding forstandard stringsfiles. It is possible to create stringsfiles using other property-list formats, including binary property-list formats and XML formats that use the UTF-8 encoding, but doing so is not recommended. For more information about the standard strings file format, see “Creating Strings Resource Files” (page 33). For more information about Unicode and its text encodings, go to http://www.unicode.org/ or http://en.wikipedia.org/wiki/Unicode. The loading of string resources (both localized and nonlocalized) ultimately relies on the bundle and internationalization supportfound in bothOS X and iOS. Forinformation about bundles,see Bundle Programming Guide . For more information about internationalization and localization,see Internationalization Programming Topics. Creating Strings Resource Files Although you can create strings files manually, it is rarely necessary to do so. The easiest way to create strings files is to write your code using the appropriate string-loading macros and then use the genstrings command-line tool to extract those strings and create strings files for you. The following sections describe the process of how to set up your source files to facilitate the use of the genstrings tool. For detailed information about the tool, see genstrings man page. Choosing Which Strings to Localize When it comes to localizing your application’s interface, it is not always appropriate to localize every string used by your application. Translation is a costly process, and translating strings that are never seen by the user is a waste of time and money. Strings that are not displayed to the user, such as notification names used internally by your application, do not need to be translated. Consider the following example: if (CFStringHasPrefix(value, CFSTR("-")) { CFArrayAppendValue(myArray, value);}; In this example, the string “-” is used internally and is never seen by the user; therefore, it does not need to be placed in a strings file. The following code shows another example of a string the user would not see. The string "%d %d %s" does not need to be localized, since the user never sees it and it has no effect on anything that the user does see. matches = sscanf(s, "%d %d %s", &first, &last, &other); String Resources Creating Strings Resource Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 33Because nib files are localized separately, you do not need to include strings that are already located inside of a nib file. Some of the strings you should localize, however, include the following: ● Strings that are programmatically added to a window, panel, view, or control and subsequently displayed to the user. This includes strings you pass into standard routines, such as those that display alert boxes. ● Menu item title strings if those strings are added programmatically. For example, if you use custom strings for the Undo menu item, those strings should be in a strings file. ● Error messages that are displayed to the user. ● Any boilerplate text that is displayed to the user. ● Some stringsfromyour application’sinformation property list(Info.plist) file;see Runtime Configuration Guidelines. ● New file and document names. About the String-Loading Macros The Foundation and Core Foundation frameworks define the following macros to make loading strings from a strings file easier: ● Core Foundation macros: ● CFCopyLocalizedString ● CFCopyLocalizedStringFromTable ● CFCopyLocalizedStringFromTableInBundle ● CFCopyLocalizedStringWithDefaultValue ● Foundation macros: ● NSLocalizedString ● NSLocalizedStringFromTable ● NSLocalizedStringFromTableInBundle ● NSLocalizedStringWithDefaultValue You use these macrosin yoursource code to load stringsfrom one of your application’sstringsfiles. The macros take the user’s current language preferences into account when retrieving the actual string value. In addition, the genstrings tool searches for these macros and uses the information they contain to build the initial set of strings files for your application. For additional information about how to use these macros,see “Loading String ResourcesInto Your Code” (page 37). String Resources Creating Strings Resource Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 34Using the genstrings Tool to Create Strings Files At some point during your development, you need to create the strings files needed by your code. If you wrote your code using the Core Foundation and Foundation macros, the simplest way to create your strings files is using the genstrings command-line tool. You can use thistool to generate a new set ofstringsfiles or update a set of existing files based on your source code. To use the genstrings tool, you typically provide at least two arguments: ● A list of source files ● An optional output directory The genstrings tool can parse C, Objective-C, and Java code files with the .c, .m, or .java filename extensions. Although not strictly required, specifying an output directory is recommended and is where genstrings places the resulting strings files. In most cases, you would want to specify the directory containing the project resources for your development language. The following example shows a simple command for running the genstrings tool. This command causes the tool to parse all Objective-C source files in the current directory and put the resulting strings files in the en.lproj subdirectory, which must already exist. genstrings -o en.lproj *.m The first time you run the genstrings tool, it creates a set of new stringsfilesfor you. Subsequent runsreplace the contents of those strings files with the current string entries found in your source code. For subsequent runs, it is a good idea to save a copy of your current strings files before running genstrings. You can then diff the new and old versions to determine which strings were added to (or changed in) your project. You can then use this information to update any already localized versions of your strings files, rather than replacing those files and localizing them again. Within a single strings file, each key must be unique. Fortunately, the genstrings tool is smart enough to coalesce any duplicate entries it finds. When it discovers a key string used more than once in a single strings file, the tool merges the comments from the individual entries into one comment string and generates a warning. (You can suppressthe duplicate entries warning with the -q option.) If the same key string is assigned to strings in different strings files, no warning is generated. For more information about using the genstrings tool, see the genstrings man page. String Resources Creating Strings Resource Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 35Creating Strings Files Manually Although the genstrings tool is the most convenient way to create strings files, you can also create them manually. To create a stringsfile manually, create a new file in TextEdit (or your preferred text-editing application) and save it using the Unicode UTF-16 encoding. (When saving files, TextEdit usually chooses an appropriate encoding by default. To force a specific encoding, you must change the save options in the application preferences.) The contents of thisfile consists of a set of key-value pairs along with optional comments describing the purpose of each key-value pair. Key and value strings are separated by an equal sign, and the entire entry must be terminated with a semicolon character. By convention, comments are enclosed inside C-style comment delimiters (/* and */) and are placed immediately before the entry they describe. Listing 2-2 shows the basic format of a strings file. The entries in this example come from the English version of the Localizable.strings file from the TextEdit application. The string on the left side of each equal sign representsthe key, and the string on the rightside representsthe value. A common convention when developing applications is to use a key name that equals the value in the language used to develop the application. Therefore, because TextEdit was developed using the English language, the English version of the Localizable.strings file has keys and values that match. Listing 2-2 Strings localized for English /* Menu item to make the current document plain text */ "Make Plain Text" = "Make Plain Text"; /* Menu item to make the current document rich text */ "Make Rich Text" = "Make Rich Text"; Listing 2-3 shows the German translation of the same entries. These entries also live inside a file called Localizable.strings, but this version of the file is located in the German language project directory of the TextEdit application. Notice that the keys are still in English, but the values assigned to those keys are in German. This is because the key strings are never seen by end users. They are used by the code to retrieve the corresponding value string, which in this case is in German. Listing 2-3 Strings localized for German /* Menu item to make the current document plain text */ "Make Plain Text" = "In reinen Text umwandeln"; /* Menu item to make the current document rich text */ "Make Rich Text" = "In formatierten Text umwandeln"; String Resources Creating Strings Resource Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 36Detecting Non-localizable Strings AppKit–based applications can take advantage of built-in support to detect strings that do not need to be localized and those that need to be localized but currently are not. To use this built-in support, you must launch your application from the command line. In addition to entering the path to your executable, you must also include the name of the desired setting along with a Boolean value to indicate whether the setting should be enabled or disabled. The available settings are as follows: ● The NSShowNonLocalizableStrings setting identifies strings that are not localizable. The strings are logged to the shell in upper case. This option occasionally generates some false positives but is still useful overall. ● The NSShowNonLocalizedStrings setting locates strings that were meant to be localized but could not be found in the application’s existing strings files. You can use this setting to catch problems with out-of-date localizations. For example, to use the NSShowNonLocalizedStrings setting with the TextEdit application, you would enter the following in Terminal: /Applications/TextEdit.app/Contents/MacOS/TextEdit -NSShowNonLocalizedStrings YES Loading String Resources Into Your Code The Core Foundation and Foundation frameworks provide macrosfor retrieving both localized and nonlocalized strings stored in strings files. Although the main purpose of these macros is to load strings at runtime, they also serve a secondary purpose by acting as markers that the genstrings tool can use to locate your application’s string resources. It is this second purpose that explains why many of the macros let you specify much more information than would normally be required for loading a string. The genstrings tool uses the information you provide to create or update your application’s strings files automatically. Table 2-1 lists the types of information you can specify for these routines and describes how that information is used by the genstrings tool. Table 2-1 Common parameters found in string-loading routines Parameter Description The string used to look up the corresponding value. This string must not contain any characters from the extended ASCII character set, which includes accented versions of ASCII characters. If you want the initial value string to contain extended ASCII characters, use a routine that lets you specify a default value parameter. (For information about the extended ASCII character set, see the corresponding Wikipedia entry.) Key String Resources Loading String Resources Into Your Code 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 37Parameter Description The name of the strings file in which the specified key is located. The genstrings tool interprets this parameter as the name of the strings file in which the string should be placed. If no table name is provided, the string is placed in the default Localizable.strings file. (When specifying a value for this parameter, include the filename without the .strings extension.) Table name The default value to associate with a given key. If no default value is specified, the genstrings tool uses the key string as the initial value. Default value strings may contain extended ASCII characters. Default value Translation comments to include with the string. You can use comments to provide clues to the translation team about how a given string is used. The genstrings tool putsthese commentsin the stringsfile and enclosesthem in C-style comment delimiters (/* and */) immediately above the associated entry. Comment An NSBundle object or CFBundleRef type corresponding to the bundle containing the stringsfile. You can use thisto load stringsfrom bundles other than your application’s main bundle. For example, you might use this to load localized strings from a framework or plug-in. Bundle When you request a string from a strings file, the string that is returned depends on the available localizations (if any). The Cocoa and Core Foundation macros use the built-in bundle internationalization support to retrieve the string whose localization matchesthe user’s current language preferences. Aslong as your localized resource files are placed in the appropriate language-specific project directories, loading a string with these macros should yield the appropriate string automatically. If no appropriate localized string resource is found, the bundle’s loading code automatically chooses the appropriate nonlocalized string instead. For information about internationalization in general and how to create language-specific project directories, see Internationalization Programming Topics. For information about the bundle structure and how resource files are chosen from a bundle directory, see Bundle Programming Guide . Using the Core Foundation Framework The Core Foundation framework defines a single function and several macrosfor loading localized stringsfrom your application bundle. The CFBundleCopyLocalizedString function provides the basic implementation for retrieving the strings. However, it is recommended that you use the following macros instead: ● CFCopyLocalizedString(key, comment) ● CFCopyLocalizedStringFromTable(key, tableName, comment) ● CFCopyLocalizedStringFromTableInBundle(key, tableName, bundle, comment) String Resources Loading String Resources Into Your Code 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 38● CFCopyLocalizedStringWithDefaultValue(key, tableName, bundle, value, comment) There are several reasons to use the macros instead of the CFBundleCopyLocalizedString function. First, the macros are easier to use for certain common cases. Second, the macros let you associate a comment string with the string entry. Third, the macros are recognized by the genstrings tool but the CFBundleCopyLocalizedString function is not. For information about the syntax of the preceding macros, see CFBundle Reference . Using the Foundation Framework The Foundation framework defines a single method and several macros for loading string resources. The localizedStringForKey:value:table: method of the NSBundle classloadsthe specified string resource from a strings file residing in the current bundle. Cocoa also defines the following macros for getting localized strings: ● NSLocalizedString(key, comment) ● NSLocalizedStringFromTable(key, tableName, comment) ● NSLocalizedStringFromTableInBundle(key, tableName, bundle, comment) ● NSLocalizedStringWithDefaultValue(key, tableName, bundle, value, comment) As with Core Foundation, Apple recommends that you use the Cocoa convenience macros for loading strings. The main advantage to these macros is that they can be parsed by the genstrings tool and used to create your application’s strings files. They are also simpler to use and let you associate translation comments with each entry. For information about the syntax of the preceding macros, see Foundation Functions Reference . Additional methods for loading strings are also defined in NSBundle Class Reference . Examples of Getting Strings The following examples demonstrate the basic techniques for using the Foundation and Core Foundation macros to retrieve strings. Each example assumes that the current bundle contains a strings file with the name Custom.strings that has been translated into French. This translated file includes the following strings: /* A comment */ "Yes" = "Oui"; "The same text in English" = "Le même texte en anglais"; String Resources Loading String Resources Into Your Code 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 39Using the Foundation framework, you can get the value of the “Yes” string using the NSLocalizedStringFromTable macro, as shown in the following example: NSString* theString; theString = NSLocalizedStringFromTable (@"Yes", @"Custom", @"A comment"); Using the Core Foundation framework, you could get the same string using the CFCopyLocalizedStringFromTable macro, as shown in this example: CFStringRef theString; theString = CFCopyLocalizedStringFromTable(CFSTR("Yes"), CFSTR("Custom"), "A comment"); In both examples, the code specifies the key to retrieve, which is the string “Yes”. They also specify the strings file (or table) in which to look for the key, which in this case isthe Custom.strings file. During string retrieval, the comment string is ignored. Advanced Strings File Tips The following sections provide some additional tips for working with strings files and string resources. Searching for Custom Functions With genstrings The genstrings tool searches for the Core Foundation and Foundation string macros by default. It uses the information in these macros to create the string entries in your project’s strings files. You can also direct genstrings to look for custom string-loading functions in your code and use those functions in addition to the standard macros. You might use custom functionsto wrap the built-in string-loading routines and perform some extra processing or you might replace the defaultstring handling behavior with your own custom model. If you want to use genstrings with your own custom functions, your functions must use the naming and formatting conventions used by the Foundation macros. The parameters for your functions must match the parameters for the corresponding macros exactly. When you invoke genstrings, you specify the -s option followed by the name of the function that correspondsto the NSLocalizedString macro. Your other function names should then build from this base name. For example, if you specified the function name MyStringFunction, your other function names should be MyStringFunctionFromTable, MyStringFunctionFromTableInBundle, and MyStringFunctionWithDefaultValue. The genstrings tool looks for these functions and uses them to build the corresponding strings files. String Resources Advanced Strings File Tips 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 40Formatting String Resources For some strings, you may not want to (or be able to) encode the entire string in a string resource because portions of the string might change at runtime. For example, if a string contains the name of a user document, you need to be able to insert that document name into the string dynamically. When creating your string resources, you can use any of the formatting characters you would normally use for handling string replacement in the Foundation and Core Foundation frameworks. Listing 2-4 shows several string resources that use basic formatting characters: Listing 2-4 Strings with formatting characters "Windows must have at least %d columns and %d rows." = "Les fenêtres doivent être composes au minimum de %d colonnes et %d lignes."; "File %@ not found." = "Le fichier %@ n’existe pas."; To replace formatting characters with actual values, you use the stringWithFormat: method of NSString or the CFStringCreateWithFormat function, using the string resource as the format string. Foundation and Core Foundation support most of the standard formatting characters used in printf statements. In addition, you can use the %@ specifiershown in the preceding example to insert the descriptive text associated with arbitrary Objective-C objects. See “Formatting String Objects” in String Programming Guide for the complete list of specifiers. One problem that often occurs during translation is that the translator may need to reorder parameters inside translated strings to account for differences in the source and target languages. If a string contains multiple arguments, the translator can insert special tags of the form n$ (where n specifies the position of the original argument) in between the formatting characters. These tags let the translator reorder the arguments that appear in the original string. The following example shows a string whose two arguments are reversed in the translated string: /* Message in alert dialog when something fails */ "%@ Error! %@ failed!" = "%2$@ blah blah, %1$@ blah!"; Using Special Characters in String Resources Just as in C, some characters must be prefixed with a backslash before you can include them in the string. These characters include double quotation marks, the backslash character itself, and special control characters such as linefeed (\n) and carriage returns (\r). "File \"%@\" cannot be opened" = " ... "; "Type \"OK\" when done" = " ... "; String Resources Advanced Strings File Tips 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 41You can include arbitrary Unicode characters in a value string by specifying \U followed immediately by up to four hexadecimal digits. The four digits denote the entry for the desired Unicode character; for example, the space character is represented by hexadecimal 20 and thus would be \U0020 when specified as a Unicode character. This option is useful if a string must include Unicode characters that for some reason cannot be typed. If you use this option, you must also pass the -u option to genstrings in order for the hexadecimal digits to be interpreted correctly in the resulting strings file. The genstrings tool assumes your strings are low-ASCII by default and only interprets backslash sequences if the -u option is specified. Note: The genstrings tool always generatesstringsfiles using the UTF-16 encoding. If you include Unicode characters in your strings and do not use genstrings to create your strings files, be sure to save your strings files in the UTF-16 encoding. Debugging Strings Files If you run into problems during testing and find that the functions and macros for retrieving strings are always returning the same key (as opposed to the translated value), run the /usr/bin/plutil tool on your strings file. A strings file is essentially a property-list file formatted in a special way. Running plutil with the -lint option can uncover hidden characters or other errorsthat are preventing stringsfrom being retrieved correctly. String Resources Advanced Strings File Tips 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 42The OS X and iOS platforms were built to provide a rich multimedia experience. To support that experience, both platforms provide plenty of support for loading and using image, sound, and video resources in your application. Image resources are commonly used to draw portions of an application’s user interface. Sound and video resources are used less frequently but can also enhance the basic appearance and appeal of an application. The following sections describe the support available for working with image, sound, and video resources in your applications. Images and Sounds in Nib Files Using Xcode, you can reference your application’s sound and image files from within nib files. You might do so to associate those images or sounds with different properties of a view or control. For example, you might set the default image to display in an image view or set the image to display for a button. Creating such a connection in a nib file saves you the hassle of having to make that connection later when the nib file isloaded. To make image and sound resources available in a nib file, all you have to do is add them to your Xcode project; Xcode then lists them in the library pane. When you make a connection to a given resource file, Xcode makes a note of that connection in the nib file. At load time, the nib-loading code looksfor that resource in the project bundle, where it should have been placed by Xcode at build time. When you load a nib file that contains references to image and sound resources, the nib-loading code caches resources whenever possible for easy retrieval later. For example, after loading a nib file, you can retrieve an image associated with that nib file using the imageNamed: method of either NSImage or UIImage (depending on your platform). In OS X you can retrieve cached sound resources using the soundNamed: method of NSSound. Loading Image Resources Image resources are commonly used in most applications. Even very simple applications use images to create a custom look for controls and views. OS X and iOS provide extensive support for manipulating image data using Objective-C objects. These objects make using image images extremely easy, often requiring only a few 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 43 Image, Sound, and Video Resourceslines of code to load and draw the image. If you prefer not to use the Objective-C objects, you can also use Quartz to load images using a C-based interface. The following sections describe the processfor loading image resource files using each of the available techniques. Loading Images in Objective-C To load images in Objective-C, you use either the NSImage or UIImage object, depending on the current platform. Applications built for OS X using the AppKit framework use the NSImage object to load images and draw them. Applications built for iOS use the UIImage object. Functionally, both of these objects provide almost identical behavior when it comesto loading existing image resources. You initialize the object by passing it a pointer to the image file in your application bundle and the image object takes care of the details of loading the image data. Listing 3-1 shows how to load an image resource using the NSImage class in OS X. After you locate the image resource, which in this case is in the application bundle, you simply use that path to initialize the image object. After initialization, you can draw the image using the methods of NSImage or passthat object to other methods that can use it. To perform the exactsame task in iOS, all you would need to do is change references of NSImage to UIImage. Listing 3-1 Loading an image resource NSString* imageName = [[NSBundle mainBundle] pathForResource:@"image1" ofType:@"png"]; NSImage* imageObj = [[NSImage alloc] initWithContentsOfFile:imageName]; You can use image objectsto open any type of image supported on the target platform. Each object istypically a lightweight wrapper for more advanced image handling code. To draw an image in the current graphics context, you would simply use one of its drawing related methods. Both NSImage and UIImage have methods for drawing the image in several different ways. The NSImage class also provides extra support for manipulating the images you load. For information about the methods of the NSImage and UIImage classes, see NSImage Class Reference and UIImage Class Reference . For more detailed information about the additional features of the NSImage class, see “Images” in Cocoa Drawing Guide . Image, Sound, and Video Resources Loading Image Resources 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 44Loading Images Using Quartz If you are writing C-based code, you can use a combination of Core Foundation and Quartz calls to load image resources into your applications. Core Foundation provides the initial support for locating image resources and loading the corresponding image data into memory. Quartz takes the image data you load into memory and turns it into a usable CGImageRef that your code can then use to draw the image. There are two ways to load images using Quartz: data providers and image source objects. Data providers are available in both iOS and OS X. Image source objects are available only in OS X v10.4 and later but take advantage of the Image I/O framework to enhance the basic image handling capabilities of data providers. When it comes to loading and displaying image resources, both technologies are well suited for the job. The only time you might prefer image sources over data providers is when you want greater access to the image-related data. Listing 3-2 shows how to use a data provider to load a JPEG image. This method uses the Core Foundation bundle support to locate the image in the application’s main bundle and get a URL to it. It then uses that URL to create the data provider object and then create a CGImageRef for the corresponding JPEG data. (For brevity this example omits any error-handling code. Your own code should make sure that any referenced data structures are valid.) Listing 3-2 Using data providers to load image resources CGImageRef MyCreateJPEGImageRef (const char *imageName); { CGImageRef image; CGDataProviderRef provider; CFStringRef name; CFURLRef url; CFBundleRef mainBundle = CFBundleGetMainBundle(); // Get the URL to the bundle resource. name = CFStringCreateWithCString (NULL, imageName, kCFStringEncodingUTF8); url = CFBundleCopyResourceURL(mainBundle, name, CFSTR("jpg"), NULL); CFRelease(name); // Create the data provider object provider = CGDataProviderCreateWithURL (url); CFRelease (url); // Create the image object from that provider. Image, Sound, and Video Resources Loading Image Resources 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 45image = CGImageCreateWithJPEGDataProvider (provider, NULL, true, kCGRenderingIntentDefault); CGDataProviderRelease (provider); return (image); } For detailed information about working with Quartz images, see Quartz 2D Programming Guide . For reference information about data providers, see Quartz 2D Reference Collection (OS X) or Core Graphics Framework Reference (iOS). Specifying High-Resolution Images in iOS An iOS app should include high-resolution versions of its image resources. When the app is run on a device that has a high-resolution screen, high-resolution images provide extra detail and look better because they do not need to be scaled to fit the space. You provide high-resolution images for each image resource in your application bundle, including icons and launch images. To specify a high-resolution version of an image, create a version whose width and height (measured in pixels) are twice that of the original. You use the extra pixels in the image to provide additional detail. When saving the image, use the same base name but include the string @2x between the base filename and the filename extension. For example, if you have an image named MyImage.png, the name of the high-resolution version would be MyImage@2x.png. Put the high-resolution and original versions of your image in the same location in your application bundle. The bundle- and image-loading routines automatically look for image files with the @2x string when the underlying device has a high-resolution screen. If you combine the @2x string with other modifiers, the @2x string should come before any device modifiers but after all other modifiers, such as launch orientation or URL scheme modifiers. For example: MyImage.png - Default version of an image resource. MyImage@2x.png - High-resolution version of an image resource for devices with Retina displays. MyImage~iphone.png - Version of an image for iPhone and iPod touch. MyImage@2x~iphone.png - High-resolution version of an image for iPhone and iPod touch devices with Retina displays. Image, Sound, and Video Resources Loading Image Resources 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 46When you want to load an image, do not include the @2x or any device modifiers when specifying the image name in your code. For example, if your application bundle included the image files from the preceding list, you would ask for an image named MyImage.png. The system automatically determines which version of the image is most appropriate and loads it. Similarly, when using or drawing that image, you do not have to know whether it is the original resolution or high-resolution version. The image-drawing routines automatically adjust based on the image that wasloaded. However, if you still want to know whether an image isthe original or high-resolution version, you can check its scale factor. If the image is the high-resolution version, its scale factor is set to a value other than 1.0. For more information about how to support high-resolution devices, see “Supporting High-Resolution Screens”. Image, Sound, and Video Resources Loading Image Resources 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 47Separating your application’s data from its code can make it easier to modify your application later. If you store the configuration data for your application in resource files, you can change that configuration without having to recompile your application. Data resource files can be used to store any type of information. The following sections highlight some of the data resource types supported by iOS and OS X. Property List Files Property list files are a way to store custom configuration data outside of your application code. OS X and iOS use property lists extensively to implement features such as user preferences and information property list files for bundles. You can similarly use property lists to store private (or public) configuration data for your applications. A property-list file is essentially a set of structured data values. You can create and edit property lists either programmatically or using the Property List Editor application (located in /Developer/Applications/Utilities). The structure of custom property-list files is completely up to you. You can use property liststo store string, number, Boolean, date, and raw data values. By default, a property list stores data in a single dictionary structure, but you can assign additional dictionaries and arrays as values to create a more hierarchical data set. For information about using property lists,see Property List Programming Guide and Property List Programming Topics for Core Foundation . OS X Data Resource Files Table 4-1 lists some additional resource file types that are supported in Mac apps. 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 48 Data Resource FilesTable 4-1 Other resource types Resource Description Type In OS X, AppleScript terminology and suite files contain information about the scriptability of an application. These files can use the file extensions .sdef, .scriptSuite, or .scriptTerminology. Because the actual AppleScript commands used to script an application are visible in userscripts and the Script Editor application, these resources need to be localized. For information on supporting AppleScript, see AppleScript Overview. AppleScript files In OS X, help content typically consists of a set of HTML files created using a standard text-editing program and registered with the Help Viewer application. (For information on how to register with Help Viewer, see Apple Help Programming Guide .) It is also possible to embed PDF files, RTF files, HTML files or other custom documents in your bundle and open them using an external application, such as Preview or Safari. For information on how to open files, see Launch Services Programming Guide . Help files Data Resource Files OS X Data Resource Files 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 49This table describes the changes to Resource Programming Guide . Date Notes Modified discussion of high-resolution image resources to include all Retina displays. 2012-06-11 2011-10-12 Updated for ARC and iOS 5. Corrected information about how you specify high-resolution image resource filenames. 2010-09-15 2010-05-25 Updated references to the Apple developer website. 2009-01-06 Added information about KVO notifications during nib loading. 2008-06-26 Updated for iOS. Clarified the process of how objects are instantiated when a nib file is loaded. 2007-09-04 Reorganized content and added new information. Changed title from "Loading Resources". 2007-02-08 2005-11-09 Corrected the misidentification of a class method as an instance method. Added “Instantiating Nibs From Memory” and the link to the NSNib class reference. 2003-07-09 2003-05-28 Section on initializing nib file objects corrected and expanded. Revision history was added to existing topic. It will be used to record changes to the content of the topic. 2002-11-12 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 50 Document Revision HistoryApple Inc. © 2012 Apple Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrievalsystem, or transmitted, in any form or by any means, mechanical, electronic, photocopying, recording, or otherwise, without prior written permission of Apple Inc., with the following exceptions: Any person is hereby authorized to store documentation on a single computer for personal use only and to print copies of documentation for personal use provided that the documentation contains Apple’s copyright notice. No licenses, express or implied, are granted with respect to any of the technology described in this document. Apple retains all intellectual property rights associated with the technology described in this document. This document is intended to assist application developers to develop applications only for Apple-labeled computers. Apple Inc. 1 Infinite Loop Cupertino, CA 95014 408-996-1010 Apple, the Apple logo, AppleScript, Cocoa, iPad, iPhone, iPod, iPod touch, Mac, Objective-C, OS X, Quartz, Safari, and Xcode are trademarks of Apple Inc., registered in the U.S. and other countries. Retina is a trademark of Apple Inc. Java is a registered trademark of Oracle and/or its affiliates. iOS is a trademark or registered trademark of Cisco in the U.S. and other countries and is used under license. Even though Apple has reviewed this document, APPLE MAKES NO WARRANTY OR REPRESENTATION, EITHER EXPRESS OR IMPLIED, WITH RESPECT TO THIS DOCUMENT, ITS QUALITY, ACCURACY, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.ASARESULT, THISDOCUMENT IS PROVIDED “AS IS,” AND YOU, THE READER, ARE ASSUMING THE ENTIRE RISK AS TO ITS QUALITY AND ACCURACY. IN NO EVENT WILL APPLE BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL,OR CONSEQUENTIAL DAMAGES RESULTING FROM ANY DEFECT OR INACCURACY IN THIS DOCUMENT, even if advised of the possibility of such damages. THE WARRANTY AND REMEDIES SET FORTH ABOVE ARE EXCLUSIVE AND IN LIEU OF ALL OTHERS, ORAL OR WRITTEN, EXPRESS OR IMPLIED. No Apple dealer, agent, or employee is authorized to make any modification, extension, or addition to this warranty. Some states do not allow the exclusion or limitation of implied warranties or liability for incidental or consequential damages, so the above limitation or exclusion may not apply to you. This warranty gives you specific legal rights, and you may also have other rights which vary from state to state. Safari CSS Visual Effects GuideContents Introduction 8 At a Glance 9 Use CSS Properties to Add Gradients, Masks, Reflections, and Filters 9 Animate Changes in CSS Properties 9 Apply 2D and 3D Transformations to Any HTML Element 10 How to Use This Document 10 Prerequisites 11 See Also 11 Using Gradients 12 Creating Linear Gradients 12 Setting the Direction of Change 13 Setting the Rate of Change 14 Creating Gradient Fades 16 Creating Radial Gradients 17 Moving the Center 18 Changing the Ending Color Location 19 Adding Color Stops 20 Creating a Radial Fade 22 Creating Repeating Gradients 23 Using a Gradient as a Border Image 24 Prior Syntax (-webkit-gradient) 26 Using Masks 28 Using an Image as a Mask 28 Using a Gradient as a Mask 32 Working with WebKit Mask Properties 35 Using Reflections 37 Adding a Reflection 37 Adjusting the Reflection’s Position 38 Masking a Reflection 39 Inner Reflections 41 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 2Using CSS Filters 43 Using CSS Filters 43 Animating CSS Transitions 45 Setting Transition Properties 46 Using Timing Functions 47 Delaying the Start 48 Setting Several Transition Properties At Once 48 Handling Intermediate States and Events 49 Animating With Keyframes 50 Creating a Keyframe Animation 51 Creating Keyframes 52 Setting Animation Properties 53 Animation Timing Functions 54 Starting Animations 55 Controlling Animation Using JavaScript 56 Handling Animation Events 58 Using 2D and 3D Transforms 61 2D Transform Functions 62 2D Translation 63 2D Rotation 63 2D Scaling 65 Setting Multiple Transforms 66 Changing the Origin 67 3D Transforms 69 Adding 3D Perspective 70 Creating a 3D Space 74 3D Transform Functions 76 Back Face Visibility 80 Using Transformation Matrices 81 2D Matrix Operations 82 3D Matrix Operations 84 Working with Transforms in JavaScript 84 Example: Animated Rotating Box Under JavaScript Control 86 Adding Interactive Control to Visual Effects 90 Using a Click or Tap to Trigger a Transition Effect 90 Controlling a 3D Transform with a Click, Touch, or Swipe 91 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 3 ContentsUsing Gestures to Scale and Rotate Elements 95 Document Revision History 99 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 4 ContentsFigures and Listings Using Gradients 12 Figure 1-1 Simple linear gradient 13 Figure 1-2 Rainbow gradient 13 Figure 1-3 Diagonal gradients 14 Figure 1-4 Setting color stop percentages 15 Figure 1-5 Solid color band and abrupt color change 15 Figure 1-6 Linear gradient fade 16 Figure 1-7 Simple radial gradient 17 Figure 1-8 Circular gradient 18 Figure 1-9 3D lighting effect 19 Figure 1-10 Closest-corner gradient fills 20 Figure 1-11 Multicolor gradient 21 Figure 1-12 Red color stop at 20% 21 Figure 1-13 Color band and abrupt color change using color stops 22 Figure 1-14 Spotlight gradient 23 Figure 1-15 Repeating gradient patterns 24 Figure 1-16 Linear gradient border 25 Figure 1-17 Radial gradient border 26 Listing 1-1 Linear fade 16 Listing 1-2 Radial fade 22 Using Masks 28 Figure 2-1 Heart-shaped “cookie-cutter” 29 Figure 2-2 Masking a border 30 Figure 2-3 Stacking masks 31 Figure 2-4 Applying a mask with a fuzzy border 32 Figure 2-5 Result of applying a gradient mask 33 Figure 2-6 Horizontal gradient mask with color stops 34 Figure 2-7 Masking text with a radial gradient 35 Listing 2-1 Stacking masked elements 30 Using Reflections 37 Figure 3-1 Reflection below a heading 38 Figure 3-2 Reflection with negative offset 39 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 5Figure 3-3 Reflection with image as mask 40 Figure 3-4 Image with reflection and gradient mask 41 Figure 3-5 Inner reflection, reflected 42 Using CSS Filters 43 Figure 4-1 CSS filters on a video 43 Listing 4-1 Applying CSS filters to HTML elements 44 Animating CSS Transitions 45 Figure 5-1 Card Flip example 45 Figure 5-2 Cubic Bezier timing function 47 Figure 5-3 Transition of two properties 49 Listing 5-1 Setting transition properties 46 Listing 5-2 Creating multiple transitions at once 47 Listing 5-3 Defining a custom timing function 48 Listing 5-4 Detecting transition end events 49 Animating With Keyframes 50 Figure 6-1 Animated text elements 50 Figure 6-2 Animation timing function control points 54 Figure 6-3 JavaScript control of animation 58 Listing 6-1 Simple keyframe animation 51 Listing 6-2 Declaring keyframes 52 Listing 6-3 Starting an animation 56 Listing 6-4 Pausing and continuing an animation 57 Listing 6-5 Handling animation events 59 Using 2D and 3D Transforms 61 Figure 7-1 HTML page with rotation and perspective transforms 61 Figure 7-2 A translated element and its offspring 63 Figure 7-3 Rotating an element 64 Figure 7-4 Scaling an element up 66 Figure 7-5 Element rotated around the top-right corner 68 Figure 7-6 3D coordinate space 69 Figure 7-7 Setting the perspective 72 Figure 7-8 Perspective origin effects 74 Figure 7-9 Text rotated relative to 3D backdrop 76 Figure 7-10 Z-axis translation in perspective 78 Figure 7-11 X and y rotation 79 Figure 7-12 Cardflip example 81 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 6 Figures and ListingsFigure 7-13 2D transformation matrix parameter positions 82 Figure 7-14 Matrix mirroring transforms 83 Figure 7-15 3D matrix parameters 84 Figure 7-16 3D transforms under JavaScript control 89 Listing 7-1 Animating 2D rotation 64 Listing 7-2 Setting multiple transforms using a list 67 Listing 7-3 Nesting 2D transforms 67 Listing 7-4 Rotating an element around the top-right corner 68 Listing 7-5 Adding a perspective slider 70 Listing 7-6 Effects of perspective origin 73 Listing 7-7 Nested 3D rotations 75 Listing 7-8 Hiding the back side of a card 80 Listing 7-9 Matrix example 82 Adding Interactive Control to Visual Effects 90 Figure 8-1 Click and tap handler 91 Figure 8-2 Page flip in action 93 Figure 8-3 Element rotated by a gesture 96 Listing 8-1 Simple touch or tap handler 90 Listing 8-2 Page flip on click, tap, or swipe 93 Listing 8-3 Responding to gesture events 96 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 7 Figures and ListingsUse CSS to create stunning visual effects—masks, gradients, reflections, lighting effects, animations, transitions, 3D rotations, and more. Apply any or all of these effects interactively, triggered by mouse events or touch events; make HTML elements visibly respond to the user—without requiring plug-ins, graphics libraries, or elaborate JavaScript programs. There are advantages to using CSS instead of graphic images to create visual effects: ● Because they are resolution-independent, CSS effects scale up smoothly when zoomed. ● Text formatted with CSS is searchable; images are not. ● CSS is compact and compresses well compared with graphic images. ● CSS is just text; it can be quickly modified using a text editor or the output of a script. Safari supports CSS visual effects on Mac OS X and iOS. 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 8 IntroductionAt a Glance Safari CSS visual effectsfall into three categories: new visual CSS properties, animation, and 2D and 3D transforms. Use CSS Properties to Add Gradients, Masks, Reflections, and Filters New visual CSS properties include gradients, masks, reflections, and filters. Gradients let you add beautiful, resolution-independent color blends to backgrounds and borders, with a single line of CSS. Use masks to render portions of HTML elements transparent for elegant compositing. Apply a mask as you would a background or a border image. You can use an image as a mask. You can also use a gradient as a mask, and you can mask any HTML element, not just images. Add a reflection to any element; use a gradient as a mask for a reflection to make the reflection fade to transparency. Filters allow you to add hardware-accelerated visual effects to HTML elements, including images and videos. Relevant chapters: “Using Gradients” (page 12), “Using a Gradient as a Mask” (page 32), “Using Reflections” (page 37), “Using CSS Filters” (page 43). Animate Changes in CSS Properties CSS makes animation easy: specify the properties you want animated, and optionally supply a duration for the animation, and any change to those CSS properties is automatically made into an animation of the HTML element, without using graphic plug-ins or even JavaScript. Use CSS pseudoclasses such as :hover to make the animations interactive—have elements fade in, grow, or enter from offscreen in response to touch or mouse events. CSS animations come in two flavors: implicit animations that render changes smoothly over a defined period, and keyframe animations that allow for more complex behavior, such as moving from side to side or starting and stopping en route. Introduction At a Glance 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 9Relevant chapters: “Animating CSS Transitions” (page 45), “Animating With Keyframes” (page 50). Apply 2D and 3D Transformations to Any HTML Element You can apply 2D or 3D transforms to any HTML element, turning a group of div elements into the faces of a box or the pages of a book, for example. Apply perspective and animation, and you can open and close the box, turn it to look inside, flip the pages of the book, and so on. 2D transforms include scaling, translation, shearing, reflection, and rotation. 3D transforms add rotation about the x and y axis and displacement on the z axis. Add touch and mouse interaction to trigger transformations by implementing CSS pseudoclasses, such as :hover, or by writing your own JavaScript functions. Relevant chapters: “Using 2D and 3D Transforms” (page 61), “Adding Interactive Control to Visual Effects” (page 90). How to Use This Document The visual effects described in this document are WebKit extensions of CSS. Most of the extensions are proposals for W3C standards;some are in W3C draftsfor CSS3. Asthe standards evolve,syntax for these effectsis modified. This change is done carefully to allow new syntax to coexist with existing syntax, however. This means you can experiment with CSS extensions without having your website suddenly break when the standard is modified—in most cases, the old syntax still works. This document describes the current syntax as of this writing; many of the extensions have prior syntax that still works but is no longer recommended. Introduction How to Use This Document 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 10Note: Because these extensions are WebKit-specific, they are not supported in all browsers. Most of the extensions have equivalentsin other WebKit-based browsers, however, and for mobile-specific sites, WebKit-based browsers account for nearly all traffic. Many of the extensions are currently in W3C working drafts and have equivalents in non-WebKit browsers as well, using the same syntax. Nevertheless, unless you are writing an iOS web app, you should code your website to degrade gracefully for browsersthat do notsupport these extensions. For the most part, thisis automatic—an image may not have a reflection in some browsers, or a change in CSS properties may be immediate instead of animated—but the site remains functional and the layout is not broken. As always, you should test your website using all the browsers that you wish to support to ensure that all of your users have an aesthetically pleasing experience. Prerequisites You need a solid understanding of HTML and some familiarity with JavaScript and CSS to make good use of this document. See Also You may also find the following documents helpful: ● Safari DOM Additions Reference—describes the touch event classes that you use to handle multi-touch gestures in JavaScript. ● Safari CSS Reference—describes the CSS properties supported by various Safari and WebKit applications. ● Safari Web Content Guide—describes how to create content that is compatible with, optimized for, and customized for iOS. ● iOS Human Interface Guidelines—provides user interface guidelines for designing webpages and web applications for Safari on iOS. ● FingerTips—demonstrates how to build an interactive 3D carousel using CSS,JavaScript and touch events. ● http://dev.w3.org/csswg/css3-images/—W3C draft for gradients. ● http://www.w3.org/TR/css3-transitions/—W3C draft for animated transitions. ● http://www.w3.org/TR/css3-animations/—W3C draft for keyframe animations. ● http://www.w3.org/TR/css3-2d-transforms/—W3C draft for 2D transforms. ● http://www.w3.org/TR/css3-3d-transforms/—W3C draft for 3D transforms. Introduction Prerequisites 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 11Use gradients as color fills that blend smoothly from one color to another. Use a CSS gradient anywhere that you can use an image, such as for the background of an element, an element border, or a mask. Because gradients are resolution-independent and compact, a line or two of CSS can replace hundreds of kilobytes—or even megabytes—of graphic imagery. Unlike graphic images, gradients have no inherentsize, and automatically expand to fill a container. To create a gradient, specify a starting color and an ending color, and optionally intermediate colors and a direction. Safari supports two types of CSS gradients: linear and radial. This chapter describes both types of gradients. Safari 5.1 on the desktop, and Safari on iOS 5.0, use the -webkit- prefix for specifying gradients, but otherwise conform to the 17 February 2011 working draft for CSS3 gradients: http://dev.w3.org/csswg/css3- images/. Note: Recent drafts of the W3C proposal have simplified the syntax. This chapter describesthe most recent implementation shipping in Safari. You should expect Safari’ssyntax for gradientsto continue to change as the W3C standard evolves. While new syntax is expected, the existing syntax—and prior syntax—should still work. The -webkit-linear-gradient and webkit-radial-gradient properties require iOS 5.0 or later, or Safari 5.1 or later on the desktop. If you need to support earlier releases of iOS or Safari, see “Prior Syntax (-webkit-gradient)” (page 26). Creating Linear Gradients A linear gradient defines a color change along a specified line. Each point on the line has a particular color. The width of the line, perpendicular to the direction of the line, extendsto the edges of the gradient’s container. You can use a linear gradient to fill any two-dimensional shape. By default, a linear gradient changes colors from top to bottom. For example: background: -webkit-linear-gradient(aqua, white) 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 12 Using Gradientsdefines a linear gradient that starts as aqua at the top of the element and ends as white at the bottom of the element. The gradient fills the element completely, as Figure 1-1 illustrates. Figure 1-1 Simple linear gradient If you specify intermediate colors between the starting and ending color, the gradient blends from color to color. For example: background: -webkit-linear-gradient(red, yellow, orange, green, blue, purple); defines a rainbow gradient as a background. Apply this style to a div element, and the element is drawn with a rainbow background, as Figure 1-2 illustrates. Figure 1-2 Rainbow gradient Setting the Direction of Change You can define a linear gradient with the color change going in any direction: from any edge or corner to its opposite edge or corner, or at any specified angle. To specify a direction from edge-to-edge or corner-to-corner, just specify the beginning edge or corner. For example: ● background: -webkit-linear-gradient(left, black, white); creates a horizontal gradient going from left to right. ● background: -webkit-linear-gradient(bottom right, black, white); Using Gradients Creating Linear Gradients 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 13creates a diagonal gradient from bottom right to top left. You can also specify color change direction by supplying an angle. Angles are given in degrees, with 0deg being straight up and proceeding counterclockwise-positive, so that 90deg is horizontal left and 180deg is straight down. For example: -webkit-linear-gradient(45deg, black, white) creates a gradient at a 45 degree angle up and to the left. Note: If you specify a gradient from corner to corner, the angle of the gradient changesif the parent element is resized and the shape of the element changes. Specify the direction in degrees to create a gradient with a fixed angle. Figure 1-3 shows a diagonal gradient starting at the bottom left corner. Figure 1-3 Diagonal gradients Setting the Rate of Change By default, the rate of color change for a gradient remains constant; if the gradient has three colors, the blend starts with the first color at 0% of the gradient length, reaches the second color at 50% of the gradient length, and reaches the third color at 100% of the gradient length. In other words, the first line of the gradient is the starting color, the middle line is the second color, and the last line is the third color. To modify this behavior, specify color stops. For example, the following snippet creates a gradient that changes gradually from white to cornflower blue over 90% of the gradient length then blends quickly from to black over the remaining 10%: -webkit-linear-gradient(left, white, cornflowerblue 90%, black) Using Gradients Creating Linear Gradients 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 14Figure 1-4 shows such a gradient. Figure 1-4 Setting color stop percentages Color stops can create some striking effects. For example, specify the same color at two consecutive stops to create a band of solid color, or specify two different colors at the same percentage point to create an abrupt change in color. Figure 1-5 shows the effect these kind of color stops create. background: -webkit-linear-gradient(left,black,blue 10%,blue 90%,black); background: -webkit-linear-gradient(left,white,blue 50%,purple 50%,white); Figure 1-5 Solid color band and abrupt color change Using Gradients Creating Linear Gradients 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 15Creating Gradient Fades Use RGBa colors in a gradient to soften or fade colors into the background by decreasing the alpha value of the gradient. For example, Listing 1-1 creates a div element with a white background that fadesto transparent. Two consecutive white color stops are used, so the div element’s background stays white for 50% of its width and then fades into the background of the element’s parent. Figure 1-6 shows the result. Listing 1-1 Linear fade rgba gradient

RGBa Gradient Fades

Figure 1-6 Linear gradient fade Using Gradients Creating Linear Gradients 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 16Creating Radial Gradients A radial gradient is a color fill that blends from one color to another with the color change proceeding radially, forming a disk clipped by the shape of the element. The disk can be a circle or an ellipse. Specify a starting and ending color, and optionally specify intermediate colors. The starting color begins at the center of the disk, and the color change proceeds outward until the ending color isreached, by default at the farthest corner of the element being filled. By default, a radial gradient is an ellipse the height and width of the element being filled, with the center of the ellipse at the center of the element being filled. For example, the following snippet creates a radial gradient that blends from white at the center of a div element to black at the element’s outer edges, as Figure 1-7 illustrates:
Figure 1-7 Simple radial gradient To create a circular gradient, pass in the circle parameter: -webkit-radial-gradient(circle, white, black). Using Gradients Creating Radial Gradients 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 17The circle parameter parameter causes the gradient to be circular, instead of an ellipse that conforms to the shape of the element, as Figure 1-8 illustrates: Figure 1-8 Circular gradient Note that the shape of an element clips a circular gradient just as it does an elliptical gradient. Moving the Center By default, the starting point of a radial gradient isthe center of the element you are filling. Change this behavior by specifying a different starting point, using horizontal or vertical offsets from the element’s top left corner. The following example creates a circular div element (a square div element with a 50% border radius), then fills it with a radial gradient whose center is offset down and to the right of the element’s upper-left corner by 30%, creating a 3D lighting effect, as Figure 1-9 illustrates:
Figure 1-9 3D lighting effect Notice that you specify the vertical and horizontal offsets are as a distance from the top and left edge of the element, and that you specify both offsets as part of a single comma-delimited parameter, separated by a space. If you specify only a single value, Safari treats it as a vertical offset. Changing the Ending Color Location By default, a gradient reaches its ending color at the corner of the element furthest from the specified center of the gradient. Modify this behavior by specifying closest-corner. For example: -webkit-radial-gradient(30% 10%, closest-corner, white, black) creates a radial gradient that begins 30% to the right and 10% below the upper-left corner of an element, and reaches its end color at the closest corner. If you specify both the circle and the closest-corner properties, passthem in the same comma-delimited parameter, separated by a space: -webkit-radial-gradient(30% 10%, circle closest-corner, white, black) Using Gradients Creating Radial Gradients 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 19When the gradient ends at the closest corner, the ending color fills the remainder of the element. Figure 1-10 shows examples of identical div elements filled with radial gradients offset 30% down and to the right of the upper-left corner. Elliptical gradients are on the left of the illustration and circular gradients are on the right. Closest-corner gradient fills are above and default gradient fills are below: Figure 1-10 Closest-corner gradient fills Adding Color Stops To create a multi-colored radial gradient, specify intermediate colors between the starting and ending color. For example, the following snippet creates a radial gradient that blends from white at the center to green half way out, to black at the outer edge, as Figure 1-11 illustrates:
Figure 1-11 Multicolor gradient By default, the rate of color change is constant, dividing the gradient into equal-size color blends. Modify this behavior by providing color stops that specify the distance from the gradient center to its edge for particular colors. For example, the following snippet creates a gradient that blends from white to red in just 20% of the gradient, then takes the remaining 80% to fade to black, as Figure 1-12 illustrates:
Figure 1-12 Red color stop at 20% Using Gradients Creating Radial Gradients 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 21Specify paired color stops of the same color to create bands of solid color, or specify pairs of color stops with different colors at the same percentage point to create abrupt color changes. The following snippet defines two radial gradients, one with two color stops of the same color and the other with two differently colored stops at the same percentage point. Figure 1-13 shows the result: background: -webkit-radial-gradient(white, red 10%, red 90%, black); background: -webkit-radial-gradient(white, yellow 20%, red 20%, black); Figure 1-13 Color band and abrupt color change using color stops Creating a Radial Fade If you use RGBa colors in your color stops, you can specify both the hue and transparency of the gradient at any point. This enables you to create a gradient that blends into the background of its parent element. For example, Listing 1-2 creates a spotlight effect by fading a div element’s background from white to transparent, allowing the parent div’s background to gradually show through. Figure 1-14 shows the result: Listing 1-2 Radial fade rgba gradient

Spotlight On: Gradients

Using RGBA Colors

A radial gradient using rgba colors can be used to create a spotlight effect.

Figure 1-14 Spotlight gradient Creating Repeating Gradients You can create a repeating pattern in a gradient in two different ways. One way isto specify a series of repeating color stops. For example, the following snippet creates linear and radial gradients with two repeating red-to-blue-to-red blend stripes: -webkit-linear-gradient(red, blue 25%, red 50%, blue 75%, red) -webkit-radial-gradient(red, blue 25%, red 50%, blue 75%, red) It’s tedious to specify the color stops repeatedly when there are many repetitions, particularly if you need to calculate cumulative percentages for each color stop. To simplify the process, use the repeating gradient properties: -webkit-repeating-linear-gradient and -webkit-repeating-radial-gradient. To create a repeating gradient, specify the first set of color stops with percentages; the gradient repeats the pattern of colorstops, keeping the percentages proportional, as needed to reach 100%. The syntax for repeating Using Gradients Creating Repeating Gradients 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 23gradients is otherwise identical to nonrepeating gradients. The following snippet specifies color stops that fill 20% of two gradients, which createslinear and radial gradients with five repeating plum-to-powderblue-to-plum blend stripes, as Figure 1-15 illustrates. -webkit-repeating-linear-gradient(plum, powderblue 10%, plum 20%) -webkit-repeating-radial-gradient(plum, powderblue 10%, plum 20%) Figure 1-15 Repeating gradient patterns Note that the color pattern repeats from the starting color, so if the last color specified is the same as the starting color, the pattern repeats smoothly. If you specify a pattern whose last entry is different from the first entry, the color changes abruptly from the last color to the first color when the pattern repeats, instead of blending smoothly. Using a Gradient as a Border Image You can use a gradient anywhere you can use an image—as you would expect, this means you can use a gradient as a border image. However, the syntax is nonobvious. The border-image property uses four values to specify offsets for slicing the image into the top, bottom, and sides of the border. Since a gradient has no inherent size, a pixel or percentage offset into the image is meaningless; the only useful value is 100%. To make tiles from a linear gradient so that the border edges have the same color everywhere the tiles meet, start and end the gradient with the same color. For example, the following snippet creates a border like the one that Figure 1-16 shows.
Figure 1-16 Linear gradient border Radial gradients work nicely as borders, because the outer edges of all the tiles typically match when the stretch value is used for the repeat parameter. Here is an example of a radial gradient used as a border image:
Using Gradients Using a Gradient as a Border Image 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 25Figure 1-17 shows the result. Figure 1-17 Radial gradient border Prior Syntax (-webkit-gradient) The -webkit-linear-gradient and -webkit-radial-gradient properties are supported in iOS 5.0 and later, and in Safari 5.1 and later on the desktop. In prior versions of Safari, both linear and radial gradients were implemented using the -webkit-gradient property. If you need to support earlier versions of iOS or Safari, use the -webkit-gradient property to create gradients. The syntax is as follows: Specify linear gradients using the keyword linear, followed by a starting point, an ending point, a starting color with the keyword from, any color stops, and an ending color with the keyword to. ● Linear gradient, vertical from top: -webkit-gradient(linear, center top, center bottom, from(white), to(black)) ● Linear gradient, horizontal from left: -webkit-gradient(linear, center left, center right, from(white), to(black)) ● Linear gradient, diagonal from upper left: -webkit-gradient(linear, upper left, lower right, from(white), to(black)) ● Linear gradient, rainbow: Using Gradients Prior Syntax (-webkit-gradient) 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 26-webkit-gradient(linear, center left, center right, from(yellow), color-stop(20%, orange), color-stop(40%, red), color-stop(60%, green), color-stop(80%, blue), to(purple)) Specify radial gradients using the keyword radial, followed by a starting point, a starting radius, an ending point, an ending radius, a starting color, any color stops, and an ending color. ● Radial gradient, centered: -webkit-gradient(radial, center center, 0px, center center, 100%, from(white), to(black)) ● Radial gradient, offset: -webkit-gradient(radial, 20% 20%, 0px, 20% 20%, 60px, from(white), to(black)) ● Radial gradient, rainbow: -webkit-gradient(radial, center center, 0px, center center, 100%, from(yellow), color-stop(20%, orange), color-stop(40%, red), color-stop(60%, green), color-stop(80%, blue), to(purple)) The beginning radius specifies the size of the disk at the center of the gradient. The ending radius specifies the size of the disk at the end of the gradient. The gradient is circular if the beginning and ending center points are the same, elliptical otherwise. The gradient is clipped by the shape of the containing element if the ending radius specifies a larger disk than the element can contain. Using Gradients Prior Syntax (-webkit-gradient) 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 27A mask is a two-dimensional shape that acts as an overlay on an element and clips the element visually or renders portions of the element transparent or translucent. A mask consists of an image or a gradient that contains an alpha channel; where the alpha value of the mask is 0, the underlying element is transparent, or masked out; where the alpha value is 1, the underlying element displays normally; where the alpha value is between 0 and 1, the underlying element is proportionally transparent. Only the alpha values of the mask are used; any other color values are ignored. An element’s mask applies to its children and their descendants. The children and descendants may also have masks, in which case the masks are stacked. The transparent areas of a masked element reveal the underlying webpage contents, which may also contain masked elements; these masks too are stacked. The final image is rendered after concatenating all the masks in the stack, tiling and stretching them as specified. When a mask is applied to an element, both the contents and background of the element are masked. A mask may be applied to the entire element or it may exclude either the border or the combined border and padding. Using an Image as a Mask To apply a mask using an image, first create a .png or .svg image that contains an alpha channel. Pass the URL of thisimage into either the -webkit-mask-image property or the -webkit-mask-box-image property. The -webkit-mask-image property usesthe native size of the mask image. The -webkit-mask-box-image property scales the mask image to meet the edges of the element. If the image used as a mask consists of an opaque shape on a transparent background, the mask acts as a “cookie cutter” to clip an element in the shape of the mask. For example, the following snippet applies a heart-shaped mask to an image. Figure 2-1 shows the result. 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 28 Using Masks Figure 2-1 Heart-shaped “cookie-cutter” The -webkit-mask-box-image property takes optional arguments that work like the -webkit-border-image parameters. You can specify offsets into the mask image to slice it into top, right, bottom, and left images, applying the mask slices, stretched or repeated, to the edges of an element. For example, the following snippet uses a 50-pixel heart-shaped mask and adds a 50-pixel border to the image. The snippet also adds parameters to the -webkit-mask-box-image property, setting the slice size to the whole mask image and specifying repeat both horizontally and vertically. Figure 2-2 shows the result. Figure 2-2 Masking a border As mentioned previously, you can mask any HTML element, and you can concatenate masks by inheritance and by stacking masked elements. For example, Listing 2-1 creates a pair of masked div elements, each of which contains a paragraph element. The example positions the second div element so that it partly overlaps the first div element. The paragraph elements are masked by their parent div elements, and the two masks are stacked. The result is shown in Figure 2-3. Listing 2-1 Stacking masked elements stacked masks

This div element has a heart-shaped mask.This div element has a heart-shaped mask.This div element has a heart-shaped mask.This div element has a heart-shaped mask.This div element Using Masks Using an Image as a Mask 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 30has a heart-shaped mask.This div element has a heart-shaped mask.This div element has a heartshaped mask.This div element has a heart-shaped mask.

This div element has a heart-shaped mask.This div element has a heart-shaped mask.This div element has a heart-shaped mask.This div element has a heart-shaped mask.This div element has a heart-shaped mask.This div element has a heart-shaped mask.This div element has a heartshaped mask.This div element has a heart-shaped mask.

Figure 2-3 Stacking masks Using Masks Using an Image as a Mask 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 31You can use a mask to fade an element gradually into the background by including semitransparent pixels in the mask. For example, the following snippet applies a mask image with a solid center and increasingly transparent edges to the image of a field. The -webkit-mask-box-image property is used, so the mask is scaled to fit the img element. The result is shown in Figure 2-4. Figure 2-4 Applying a mask with a fuzzy border Image Mask Result Using a Gradient as a Mask Because gradients can be used in place of any image, a gradient can be passed as a mask. The following snippet defines a gradient that goes from opaque to transparent as a mask for an img element. The result is shown in Figure 2-5. Figure 2-5 Result of applying a gradient mask Note: The gradient in the example uses black asthe opaque color, but any completely opaque color would be equivalent. The example just given goesfrom opaque to transparent over the height of the image, but you can use different gradient parameters to achieve different effects. The following snippet creates a gradient from left to right and uses color stops to fade an image sharply at the edges, as shown in Figure 2-6. Notice that the image’s border radius works in concert with the mask. Figure 2-6 Horizontal gradient mask with color stops You can apply a gradient mask to any HTML element, not just images. For example, the following snippet creates a radial gradient as a mask for a div element containing several paragraphs of text. The result is shown in Figure 2-7.
Figure 2-7 Masking text with a radial gradient For more about gradients, see “Using Gradients” (page 12). Working with WebKit Mask Properties You can use either the -webkit-mask-image or the -webkit-mask-box-image property to assign a mask to an element. The -webkit-mask-box-image property scales the mask to fit the element by default, while the -webkit-mask-image property uses the mask image’s native size. Note: Gradients do not have a native size, so when using a gradient as a mask, the mask is always scaled to fit the mask bounding area (by default, the same as the masked element’s bounding area). The -webkit-mask-box-image property accepts optional parameters that enable it to act like a border image (for details, see Figure 2-5 (page 33)). There is also a set of propertiesthat you can use to modify the behavior of the -webkit-mask-image property, including: ● -webkit-mask-clip—Causes the mask to apply to the whole element, only the padding and content, or only the content. Set to border-box (default), padding-box, or content-box. Using Masks Working with WebKit Mask Properties 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 35● -webkit-mask-origin—Sets the origin of the mask to the upper left corner of the element’s border, padding, or content box. Set to border (default), padding, or content. ● -webkit-mask-position—Anchors the upper-left corner of the mask at the specified offset from the origin. Accepts an x-offset and a y-offset. ● -webkit-mask-repeat—Controls how the mask is tiled to cover the element. Accepts an x-repeat and a y-repeat parameter. Both parameters can be set to repeat, space, round, or no-repeat. The specifics are the same asfor the background-repeat property (see http://www.w3.org/TR/css3-background/#thebackground-repeat). ● -webkit-mask-size—Controls how the mask is sized to cover the element. The specifics are the same as for the background-size property (see http://www.w3.org/TR/css3-background/#the-backgroundsize). The properties that apply to webkit-mask-image are analogous to the CSS3 background properties. See http://www.w3.org/TR/css3-background/#backgrounds for details. Using Masks Working with WebKit Mask Properties 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 36A reflection is a mirror image, optionally with its own mask. You can add a reflection to any visible HTML element. Reflections update automatically as the element’s appearance changes. If you hover over a link with a reflection, for example, you see the hover effect in the reflection. If you add a reflection to a video element, the video plays in the reflection as well. Resize the element and the reflection is resized. Reflections are not interactive elements, however—clicking or touching a reflection does not generate any events. Reflections have no effect on layout other than being part of a container’s overflow, and are similar to box shadows in this respect. In other words, you must position elements explicitly to allow space for reflections; if you add a reflection below an image, and follow the image with a paragraph of text, the reflection may cover the text (depending on the size of the image, padding, margins, and so on). Adding a Reflection Use the -webkit-box-reflect property to add a reflection to an element. The first, mandatory, parameter is the direction in which the reflection projects, which can be above, below, left, or right. For example, the following snippet adds a reflection below an h1 element, as Figure 3-1 illustrates:

Reflection in Blue 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 37 Using Reflections

Figure 3-1 Reflection below a heading The reflection is the same visible size as the element. The element border and padding are included in the reflection. Adjusting the Reflection’s Position The first argument of the -webkit-box-reflect property specifiesthe direction of the reflection. An optional second argument specifies the offset or space between the original image and the reflection. By default, the reflection begins with no separation from the element’s bounding box. To change this behavior, add an offset parameter. A positive offset causes the reflection to be separated from the element by additional space; a negative offset causes the reflection to overlap the element. For example, the bounding box of an h1 element includes substantial white space; to tuck the reflection in closer to the text, add a negative offset. The following example adds an offset of -15px to the reflection. The result is shown in Figure 3-1: Using Reflections Adjusting the Reflection’s Position 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 38

Figure 3-2 Reflection with negative offset Masking a Reflection An optional third argument to -webkit-box-reflect specifies a mask to apply to the reflection. A mask renders part of the reflection transparent or translucent. The mask is either an image file, specified as a URL, or a gradient. A mask image file can be a .png, an .svg, or a .gif with transparent areas. The alpha value of the mask is applied to the reflection on a pixel-by-pixel basis. Only the alpha value of the mask’s color values has any effect on the reflection. When creating a mask, shape it to mask the target element itself, not to mask the reflection. The reflection shows a mirror image of the masked element, even though the element does not appear masked. The following snippet adds a mask image to a reflection. Figure 3-3 shows the resulting masked reflection side-by-side with the mask image. Using Reflections Masking a Reflection 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 39style="-webkit-box-reflect: below 0px url(mask.png);" Figure 3-3 Reflection with image as mask The following snippet adds a reflection below an image, using a gradient as a mask. Figure 3-4 shows the result. Figure 3-4 Image with reflection and gradient mask Note that the reflection includes the rounded borders of the img element. Notice also that the gradient goes from transparent at the top to opaque at the bottom—the opposite of its appearance in the reflection; this is because the reflection shows a mirror image of the masked element. For more about masks, see “Using Masks” (page 28); for more about gradients, see “Using Gradients” (page 12). Inner Reflections When you reflect an element, all of the element’s descendants are displayed in the reflection. If you add a reflection to a child element, the child’sreflection isrendered first; thisinner reflection isincluded in the parent element’s reflection. The following snippet adds a reflection to a div element containing an img element that has its own reflection. Figure 3-5 shows the result:
Using Reflections Inner Reflections 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 41

Text...

Text...

Figure 3-5 Inner reflection, reflected Using Reflections Inner Reflections 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 42Safari 6 and later supports CSS filters, or special visual effects, that you can apply to many elements, including videos (see Figure 4-1 (page 43)). These hardware-accelerated filters (such as brightness, contrast, saturation, and blur) can be stacked on top of and animated against one another. Read “CSS Property Functions” in Safari CSS Reference to find out more about CSS filters. Figure 4-1 CSS filters on a video Using CSS Filters To add a CSS filter to an HTML element, include the -webkit-filter property in the element’s CSS declaration, as shown in Listing 4-1 (page 44). You can list as many of the functions found in “CSS Property Functions” in Safari CSS Reference as you’d like. 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 43 Using CSS FiltersListing 4-1 Applying CSS filters to HTML elements Filters

Video with and without CSS filters

The video on the left does not have CSS filters applied, while the video on the right does.

These videos are playing from the same source file.

Using CSS Filters 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 44You can create animations entirely in CSS, with no need for plug-ins, graphics libraries, or elaborate JavaScript programs. Normally, when the value of a CSS property changes, the affected elements are re-rendered immediately using the new property value. If you set CSS transition properties, however, any changes in the values of the specified CSS properties are automatically rendered as animations. This kind of automatic animation is called a transition. You trigger a transition simply by changing any of the specified CSS values. CSS property values can be changed by a pseudoclass, such as :hover, or by using JavaScript to change an element’s class name or to change its individual CSS properties explicitly. The animation flows smoothly from the original state to the changed state using a transition timing function over a specified duration. For more complex animationsthat can use arbitrary intermediate states and trigger conditions,see “Animating With Keyframes” (page 50). Transitions are especially powerful if you combine them with 2D and 3D transforms. For example, Figure 5-1 shows the results of applying an animated transition to an element as it rotates in 3D. See the CardFlip sample code project for the complete source code for this example. Figure 5-1 Card Flip example See the HTML version of this document to view the video. document to view the video. Animated transitions are a W3C draft specification: http://www.w3.org/TR/css3-transitions/. 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 45 Animating CSS TransitionsNote: Not all CSS properties are animatable. In general, any CSS property that accepts values that are numbers, lengths, percentages, or colorsis animatable. Some CSS propertiesthat accept discrete values can be animated as well, but display a jump between values rather than a smooth transition when changed, such as the visibility property. See Safari CSS Reference for which properties are animatable. Setting Transition Properties To create an animated transition, first specify which CSS properties should be animated, using the -webkit-transition-property property, a CSS property that takes other CSS properties as arguments. Set the duration of the animation using the -webkit-transition-duration property. For example, Listing 5-1 creates a div element that fades out slowly over 2 seconds when clicked. Listing 5-1 Setting transition properties

Click me and I fade away. . .

There are two special transition properties: all and none: ● -webkit-transition-property: all; ● -webkit-transition-property: none; Setting the transition property to all causes all changes in CSS properties to be animated for that element. Setting the transition property to none cancels transition animation effects for that element. To set up an animation for multiple properties, pass multiple comma-separated parameters to -webkit-transition-property and -webkit-transition-duration. The order of the parameters determines which transition the settings apply to. For example, Listing 5-2 defines a two-second -background-color transition and a four-second opacity transition. Animating CSS Transitions Setting Transition Properties 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 46Listing 5-2 Creating multiple transitions at once div.zoom-fade { -webkit-transition-property: background-color, opacity; -webkit-transition-duration: 2s, 4s; } Using Timing Functions Timing functions allow a transition to change speed over its duration. Set a timing function using the -webkit-transition-timing-function property. Choose one of the prebuilt timing functions—ease, ease-in, ease-out, or ease-in-out—or specify cubic-bezier and pass in control parameters to create your own timing function. For example, the following snippet defines a 1-second transition when opacity changes, using the ease-in timing function, which starts slowly and then speeds up: style="-webkit-transition-property: opacity; -webkit-transition-duration: 1s; -webkit-timing-function: ease-in;" Using the cubic-bezier timing function, you can, for example, define a timing function thatstarts outslowly, speeds up, and slows down at the end. The timing function is specified using a cubic Bezier curve, which is defined by four control points, as Figure 5-2 illustrates. The first and last control points are always set to (0,0) and (1,1), so you specify the two in-between control points. The points are specified using x,y coordinates, with x expressed as a fraction of the overall duration and y expressed as a fraction of the overall change. Figure 5-2 Cubic Bezier timing function Fraction of elapsed time Fraction of change 0, 0 1.0,1.0 0.9, 0.4 0.5, 0.2 0.5 0.0 1.0 0.0 0.5 1.0 Animating CSS Transitions Using Timing Functions 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 47For example, Listing 5-3 creates a 2-second animation when the opacity property changes, using a custom Bezier path. Listing 5-3 Defining a custom timing function div.zoom-fade { -webkit-transition-property: opacity; -webkit-transition-duration: 2s; -webkit-transition-timing-function: cubic-bezier(0.5, 0.2, 0.9, 0.4); } In the example just given, the custom timing function starts very slowly, completing only 20% of the change after 50% of the duration, and 40% of the change after 90% of the duration. The animation then finishesswiftly, completing the remaining 60% of the change in the remaining 10% of the duration. Delaying the Start By default, a transition animation begins as soon as one of the specified CSS properties changes. To specify a delay between the time a transition property is changed and the time the animation begins, use the -webkit-transition-delay property. For example, the following snippet defines an animation that beings 100ms after a property changes: .delay-fade { -webkit-transition-property: opacity; -webkit-transition-duration: 1s; -webkit-transition-delay: 100ms; } Setting Several Transition Properties At Once You can set an animation’s transition properties, duration, timing function, and delay using a single shorthand property: -webkit-transition. For example, the following snippet uses the :HOVER pseudostyle to cause img elements to fade in and out when hovered over, using a one-second animation that begins 100 ms after the hover begins or ends, and using the ease-out timing function: Handling Intermediate States and Events When applying a transition to an element’s property, the change animates smoothly from the old value to the new value and the property values are recomputed over time. Consequently, getting the value of a property during a transition may return an intermediate value that is the current animated value, not the old or new value. For example, suppose you define a transition for the left and background-color properties and then set both property values of an element at the same time. The element’s old position and color transition to the new position and color over time as illustrated in Figure 5-3. Querying the properties in the middle of the transition returns an intermediate location and color for the element. Figure 5-3 Transition of two properties Starting appearance Time = 0.0 sec Intermediate appearance Time = 0.5 sec Final appearance Time = 1.0 sec To determine when a transition completes, set a JavaScript event listener function for the DOM event that is sent at the end of a transition. The event is an instance of WebKitTransitionEvent, and its type is webkitTransitionEnd. For example, the snippet in Listing 5-4 displays an alert panel whenever a transition ends. Listing 5-4 Detecting transition end events document.body.addEventListener( 'webkitTransitionEnd', function(event) { alert( "Finished transition!" ); }, false ); Animating CSS Transitions Handling Intermediate States and Events 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 49Safarisupportstwo types of CSS animation: transition animations and keyframe animations. To simply animate changesin CSS properties whenever those properties change, you should probably use an animated transition (see “Animating CSS Transitions” (page 45)). To create complex animations—such as a bumblebee’s flight or a change that starts and stops or repeats indefinitely—or to trigger animations under arbitrary conditions using JavaScript, use keyframe animations. Keyframe animations include the following features: ● Choose any number of CSS properties to change and define points along a timeline that have specific states. ● Animation between two defined points is automatic but can be guided by specifying a timing function. ● You trigger keyframe animations explicitly. ● Keyframe animations can be set to repeat a finite number of times or to repeat indefinitely, proceeding in the same direction each time, or alternating forward and backward. ● Keyframe animations can be paused and resumed. ● All of the elements in a class can be animated as part of a single animation. When a keyframe animation completes, an animated element returns to its original CSS style. You can override this behavior, however, and make the final animation state persistent. You can also change an element’s style when the animation ends by installing a JavaScript listener function for the webkitAnimationEnd event. Create stunning effects by combining animation with 2D and 3D transforms. For example, Figure 6-1 shows an animation of text elements in 3D space. See the PosterCircle sample code project for the complete source code for this example. Figure 6-1 Animated text elements See the HTML version of this document to view the video. document to view the video. 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 50 Animating With KeyframesKeyframe animation is a W3C draft. See http://dev.w3.org/csswg/css3-animations/. Creating a Keyframe Animation A keyframe animation has—at minimum—a name, a duration, and one or more keyframes. A keyframe is a CSS rule consisting of a set of properties and values associated with a point in time. To create a keyframe animation, perform the following steps: 1. Create a named set of keyframes in CSS using the @-webkit-keyframes rule. The set must include at least one keyframe. 2. Set the -webkit-animation-duration property to a positive, nonzero value. 3. Set the -webkit-animation-name property to the name of the set of keyframes. The following example defines a single keyframe in a set named glow. The keyframe specifies a blue background color. The example definesthe div:hover pseudoclass as having the animation name glow and the animation duration of 1s. When the user hovers over or touches any div element, it turns blue for one second. Listing 6-1 Simple keyframe animation Blue Glow

Hover turns me briefly blue.

Animating With Keyframes Creating a Keyframe Animation 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 51

Me too.

animated div

Click to slide right and down, then back.

Controlling Animation Using JavaScript Control a CSS animation using JavaScript either by changing an element’s className property or by changing an element’s animation style properties individually. One thing to remember is that the JavaScript name for a property is not always the same as the CSS name. Compound names in CSS typically contain dashes, but in Animating With Keyframes Controlling Animation Using JavaScript 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 56JavaScript the dash is the subtraction operator, so JavaScript names omit the dash. Compound names in JavaScript usually capitalize the first character of each word after the first word; so if a CSS name were -extension-foo-bar, the JavaScript name would usually be extensionFooBar. As an example, to set the CSS property -webkit-animation-play-state from JavaScript,set the element’s style.webkitAnimationPlayState property. Listing 6-4 creates three buttons and a round div element with a radial gradient background. The first button animates the div element by changing its class name. The second and third buttons pause and continue the animation by setting the div element’s style.webkitAnimationPlayState property. Figure 6-3 illustrates the result. Listing 6-4 Pausing and continuing an animation bouncing div Animating With Keyframes Controlling Animation Using JavaScript 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 57

 

Figure 6-3 JavaScript control of animation Handling Animation Events Three animation-related events are available through the DOM event system: Animating With Keyframes Handling Animation Events 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 58● webkitAnimationStart—Sent when the animation begins (after any specified delay). ● webkitAnimationIteration—Sent at the end of each iteration. ● webkitAnimationEnd—Sent after the final iteration. Note that when the final iteration in a series completes, it triggers both a webkitAnimationIteration and a webkitAnimationEnd event. Each of these events has three significant properties: ● The event.target property is the object being animated. ● The event.animationName property is the name of the animation that generated the event. ● The event.elapsedTime property is the length of time the animation has been running. Elapsed time does not include any time the animation was in the paused play state, nor does it include any specified delay time. As an example of event handling, Listing 6-5 creates three colored balls (round div elements) that roll away and disappear when clicked. The balls are given a class name that includes animation but begins with the animation play state set to paused; an onclick handler in each ball sets the play state for that element to running. The example adds an event listener for webkitAnimationEnd events on the body element. When a webkitAnimationEnd event occurs, the listener function removes the animated element by changing the element’s style.display property to none. Listing 6-5 Handling animation events rolling divs Click the colored balls to make them roll away:
Note: To see the example in action, copy the listing into a text editor and save it with the .html file extension. Animating With Keyframes Handling Animation Events 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 60Use CSS transform propertiesto give webpages a rich visual appearance without needing image files. Elements can be positioned, rotated, and scaled in 2D and 3D space; perspective can also be applied, giving elements the appearance of depth. For example, Figure 7-1 shows a simple HTML document, containing only div elements and text but rendered using CSS transform and gradient properties. It appears to be a graphics-intensive page, yet the actual content is less than 2 KB of text, and the elements animate smoothly in 3D under user control. Figure 7-1 HTML page with rotation and perspective transforms 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 61 Using 2D and 3D TransformsHow does this work? HTML elements are, after all, inherently two-dimensional, or planar; they have height and width, but no thickness. By rotating planar elements into the third dimension and applying perspective, however, these elements can be combined to create apparently solid objects. For example, five div elements are combined in Figure 7-1 to form the sides and bottom of an open box; the ball inside the box is another div element with rounded borders—a radial gradient gives it the appearance of depth. Safari uses a series of transformation matrices to determine the mapping of every pixel on the screen. You don’t need to understand matrices to use them, however. You can apply a transform either by using a matrix or by calling one of the transform functions, such as scale() or rotate(). Transforming an element typically causes its image to be rendered in a different position on the screen, but the position and dimensions of the element on the page are not changed—the top, left, height, and width properties, for example, are not altered by transforms. It isthe coordinate system in which the element is drawn that is changed. Consequently, changing transform properties does not affect the layout of a webpage. This means that transforming an element can cause it to visually overlap neighboring elements, even though their positions and dimensions on the page may not overlap. Note: Mouse and touch events, such as onclick, track the visual representation of an element, taking 2D and 3D transforms into account. By default, transforms are applied using the center point of an element as the origin; rotation spins an object about its center, for example, and scaling expands or contracts an element from the center point. You can change the origin by setting the -webkit-transform-origin property. A transform can cause part of an element to be displayed in the element’s overflow area. If the value of the overflow property is scroll or auto, scroll bars appear as needed if a transform renders part of an object outside the display area. Safari supports both 2D and 3D transforms. Both 2D and 3D transforms are W3C drafts. See http://www.w3.org/TR/css3-2d-transforms/ and http://www.w3.org/TR/css3-3d-transforms/ for the specifications. 2D Transform Functions To apply a 2D transform to an element, use the -webkit-transform property. The transform property can be set using predefined transform functions—translation, rotation, and scaling—or it can be set using a matrix. Using 2D and 3D Transforms 2D Transform Functions 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 622D Translation 2D translation shifts the contents of an element by a horizontal or vertical offset without changing the top or left properties. The element’s position in the page layout is not changed, but the content is shifted and a shifted coordinate system applies to all descendants of the translated element. For example, if a div element is positioned at the point 10,10 using the CSS properties top and left, and the element is then translated 100 pixels to the right, the element’s content is drawn at 110,10. If a child of that div is positioned absolutely at the point 0,100, the child is also shifted to the right and is drawn at the point 110,110; the specified position of 0,100 is shifted right by 100, and the child is drawn at 100,100 relative to the parent’s upper-left corner, which is still at 10,10. Figure 7-2 illustrates this example. Figure 7-2 A translated element and its offspring To apply a 2D translation, set the -webkit-transform property to translateX(offset), translateY(offset), or both. For example:
2D Rotation 2D rotation is a rotation in the xy plane. By default, 2D rotation spins an object around its center point. To rotate an element around a different point,see “Changing the Origin” (page 67). Rotation isspecified in degrees clockwise from the element’s orientation after any inherited rotation; rotation affects the specified element and all of its descendants. The coordinate system of any descendants is likewise rotated. Using 2D and 3D Transforms 2D Transform Functions 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 63The following snippet rotates a div element 45 degrees clockwise, as shown in Figure 7-3. The div element has a beige background and contains a paragraph of text and an image; both the text and the image inherit the div element’s rotation. A second div is positioned under the rotated div to show the original div’s position prior to rotation.
Figure 7-3 Rotating an element Rotation can be specified in positive or negative degrees. For example, -webkit-transform: rotate(-45deg); specifies a 45 degree rotation counterclockwise. If rotation is animated,specifying a degree of rotation greater than the current degree causes clockwise rotation; specifying a degree of rotation less than the current degree causes counter-clockwise rotation. When animating rotation, it can be useful to specify a rotation of more than 360 degrees. For example, Listing 7-1 uses JavaScript to set a rotation of 3600deg, causing a div element to spin clockwise ten times. The text spins once on page load, and a button lets the user spin it again. Listing 7-1 Animating 2D rotation spin-in text

Spinning text!

Whew! I'm dizzy...
Notice that the CSS property -webkit-transform is addressed in JavaScript as element.style.webkitTransform. Notice also that the spin() function increments the rotation angle by 3600 each time it is called; setting the angle to 3600deg repeatedly would have no effect. 2D Scaling 2D scaling makes an element smaller or larger in one or two dimensions. Scaling affects the whole element, including any border thickness. By default, the element isscaled up or down relative to its center, which causes all four of the element’s corners to be redrawn at new locations. The element’s top, left, height, and width properties are unchanged, however,so the layout of the page is not affected. Consequently,scaling an element up can cause it to cover other elements on the page unless you design the layout to allow room for the expansion. Using 2D and 3D Transforms 2D Transform Functions 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 65Scaling modifies the coordinate system of an element’s descendants, multiplying the x and y values by the specified scaling factor. For example, if a div element contains an image positioned absolutely at 10,10, with a height and width of 100 pixels, scaling-up the div element by a factor of two results in a 200 x 200 image, positioned at 20,20 relative to the div’s upper-left corner, which moves up and to the left. Figure 7-4 illustrates this behavior. Figure 7-4 Scaling an element up Apply a scale transformation by setting the -webkit-transform property to scale(x y), where x and y are independent scale factors for width and height, or by setting the transform property to scale(size), where size is the scaling factor in both dimensions. For example: style="webkit-transform: scale(1, 2)" renders an element the same width, but twice as tall. style="webkit-transform: scale(2, 0.5)" renders an element twice as wide and half as tall. style="webkit-transform: scale(1.5)" renders an element 1.5 times larger. Setting Multiple Transforms Different transforms, such as rotation and scaling, are applied by setting different values to a single property: -webkit-transform. Consequently, if you apply one transform to an element and then specify another transform, the first transform is no longer applied; the new value overwrites the old one, as with any CSS property. There are two different ways to perform multiple transforms on an element—both scaling and rotating an element for example: Using 2D and 3D Transforms Setting Multiple Transforms 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 66● Use inheritance to apply multiple transforms: create a scaled div element, for example, add your element as a child, then rotate the child element. ● Set the -webkit-transform property of the element to a space-delimited list of transform functions, such as: -webkit-transform: scale(2) rotate(45deg); When a list of functionsis provided, the final transformation value for the element is obtained by performing a matrix concatenation of each entry in the list. (Matrix concatenation can have some side effects, such as normalizing the rotation angle modulo 360.) Both approaches—transform inheritance and transform function lists—are valid. The following two examples illustrate two ways to apply a set of transforms to an element. Listing 7-2 sets an element’s transform property to a list of transform functions. Listing 7-3 produces the same results by applying each transform to a nested element. Listing 7-2 Setting multiple transforms using a list
Listing 7-3 Nesting 2D transforms
Changing the Origin By default, the origin for transforms is the center of an element’s bounding box. Most HTML and CSS entities use the upper-left corner as the default origin, but for transforms, it is usually more convenient to use the center of an element as a reference point. Consequently, elements rotate around their center by default, and scale up or down from the center out. Using 2D and 3D Transforms Changing the Origin 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 67To change the origin for transforms of a given element, set the -webkit-transform-origin property. The new origin is specified as a distance or percentage from the element’s upper-left corner. For example, the default center origin can be expressed as -webkit-transform-origin: 50% 50%;. Changing the origin to 0% 0% or 0px 0px causes transformation to occur around the upper-left corner of the element. The code in Listing 7-4 rotates the second box in Figure 7-5 around the top-right corner. Listing 7-4 Rotating an element around the top-right corner

I am rotated about my top-right corner.

Figure 7-5 Element rotated around the top-right corner Using 2D and 3D Transforms Changing the Origin 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 683D Transforms The standard HTML coordinate system has two axes—the x-axis increases horizontally to the right, and the y-axis increases vertically downwards. With 3D transforms, a z-axis is added, with positive values rising out of the window toward the user and negative values falling away from the user, as Figure 7-6 illustrates. Figure 7-6 3D coordinate space – Z - Y + Y + X – X + Z 3D transforms move an element out of the usual xy plane, where z=0—the plane of the display. A transformed element isstill two dimensional, but it no longer liesin the usual plane. A transformed object may be translated along the z-axis, rotated around the x- or y-axis, or transformed using some combination of translation and rotation. All HTML elements have a z-index. The z-index controls the rendering order when elements overlap. An element’s z-index has nothing to do with its z-axis coordinate. Transformed objects follow the standard HTML rendering rules—objects with higher z-index values are drawn on top of other objects with lower z-index values—but for elements sharing the same z-index, the areas with higher z-axis coordinate values are drawn on top. Using 2D and 3D Transforms 3D Transforms 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 69Adding 3D Perspective To render elements with the appearance of depth, you must specify a perspective. If you apply 3D transforms without setting the perspective, elements appear flattened. For example, if you rotate an element around its y-axis without setting the perspective, the element just appears narrower. If you rotate an element 90 degrees from the default xy plane, it is seen edge-on—the element either disappears entirely or is displayed as a line. Adding perspective distorts the appearance of objects realistically, making nearby things appear larger and distant things look smaller. The closer the object, the greater the distortion. In order for Safari to create the illusion of depth, it’s necessary to specify a point of view, or perspective. Once Safari knows where the user’s eye is relative to an element, it knows how much distortion to apply and where. Use the -webkit-perspective property to set the perspective for all the descendants of an element. For example:
The perspective is specified in distance from the screen. You may specify the distance in pixels, centimeters, inches, or any CSS distance unit. If no unit type is supplied, px is assumed. Note: Perspective distortion can cause part or all of an element to be displayed offscreen, particularly if an object’s z-coordinates are positive and the projected image would extend beyond the specified viewpoint, somewhere behind the user’s eye. Tip: Perspective settings of 300px or less tend to create intense perspective distortion; settings from 500px to 1000px create moderate perspective distortion, and settings of more than 2000px give very mild perspective distortion. Listing 7-5 sets the -webkit-perspective property using a slider. Listing 7-5 Adding a perspective slider applying perspective

Left

Center

Right

The center element is edge-on, and not visible.

Using 2D and 3D Transforms 3D Transforms 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 73 Figure 7-8 Perspective origin effects Creating a 3D Space By default, the descendants of an element are flattened into the plane of their parent. When you apply a 3D transform to an element, that element’s plane is no longer the default xy plane—the plane of the display. All descendants of the element share that element’s new plane. In order to further transform the children of an element relative to that element’s plane, you must set the -webkit-transform-style property to preserve-3d, creating a 3D space. For example:
Setting the transform style to preserve-3d in an element makes that element into a 3D container; all the element’s immediate children can be manipulated independently in 3D, relative to the parent. Because HTML elements are flat, a transformed child also occupies a plane in 3D space. Each child can occupy a separate plane, or multiple children can share the same plane. By default, any descendants of these transformed children are flattened into their parental plane; setting the transform style to preserve-3d affects only an element’s immediate children. Using 2D and 3D Transforms 3D Transforms 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 743D containers can be nested. Enabling 3D transforms in one of a container element’s descendants creates a nested 3D layer; children of that descendant can be transformed in 3D, relative to their container’s plane. You need to enable 3D in a particular element only if the element’s children are to be transformed in 3D relative to that element’s plane. Any transform applied to a 3D container is inherited by all of its descendants. By applying a rotation to the highest level 3D container, for example, you are able to rotate the view of all of the container’s contents at once. Listing 7-7 gives an example of a nested pair of 3D containers, illustrated in Figure 7-9. The topmost container has a child div element rotated 45 degrees on its x-axis, so it appears to be tilted away from the viewer. This child div is also a 3D container, containing a paragraph of text rotated 35 degrees on its right edge away from the container, causing the text to appear to lift off the page. Listing 7-7 Nested 3D rotations

And so the text seems to lift off the page and float, as if transformed. Well, I suppose it is transformed, and that explains the appearance, if not quite the magic, of it...

Using 2D and 3D Transforms 3D Transforms 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 75 Figure 7-9 Text rotated relative to 3D backdrop Generally speaking, you need to create only a single 3D container—all 3D transforms can be applied relative to the default xy plane, and global transforms can be applied to the top-level container and inherited by all its descendants. Sometimesit may be convenient to manipulate a subgroup of transformed elements as a unit, however; in such a case, it makes sense to create a nested container. To disable 3D for an element’s children dynamically, set the -webkit-transform-style property to flat. Applying a 3D transform when the transform style is set to flat does not move the element out of its parent’s plane. Note: Elements that have overflow set to hidden are unable to render their child elements in 3D, and are rendered as though the transform style were set to flat. 3D Transform Functions Like 2D transforms, 3D transforms are set using the -webkit-transform property. You can apply a transform to an element by specifying a transform function, a list of transform functions, or by passing in a 3D matrix. There are several functions that perform 3D transforms: Using 2D and 3D Transforms 3D Transforms 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 76● translateZ(distance)—Moves an element closer or farther away. ● translate3d(x, y, z)—Moves an element in three dimensions. ● rotateX(degrees)—Rotates an element around the x-axis, moving the top and bottom closer or farther away. ● rotateY(degrees)—Rotates an element around the y-axis, moving the left and right sides closer or farther away. ● perspective(distance)—Sets the 3D perspective for a single element. Important: You must set a perspective for 3D transforms to have a visible 3D effect. See “Adding 3D Perspective” (page 70) and “Creating a 3D Space” (page 74) for details. 3D Translation 3D translation moves an element closer to or farther from the viewer by changing its position on the z-axis. Use the translateZ() function to shift an element on the z-axis, or the translate3d(x, y, z) function to shift an element on two or three axes. For example:
Using 2D and 3D Transforms 3D Transforms 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 77The 3D translation functions work like their 2D counterparts, except that the z offset cannot be specified as a percentage. Z-axis units may be positive (towards the viewer) or negative (away from the viewer). Figure 7-10 shows two identical div elements with same height, width, and x and y positions, one translated on the z-axis by 100 px and the other translated by -100 px. Figure 7-10 Z-axis translation in perspective All descendants of an element inherit its z-axis translation. Note that the text in the previous illustration is translated along with its parent. 3D Rotation You can rotate an element in 3D either around the y-axis,so that the right and left edges get nearer and farther away, or around the x-axis, so that the top and bottom edges get nearer and farther away. For example:
Using 2D and 3D Transforms 3D Transforms 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 78Rotation is around an imaginary x- or y-axis that passes through the element’s origin. By default, the origin is at the center of an object. Positive x units rotate the top edge away. Positive y units rotate the right edge away. This is illustrated in Figure 7-11. Figure 7-11 X and y rotation All descendants of an element inherit its 3D rotation. Note that the text in the previous illustration is rotated along with its parent. Setting Perspective for a Single Element To create a 3D space with a shared perspective, you need to create a 3D container that has the -webkit-perspective property set; but if you just want to render a single element with the appearance of depth, you can set the -webkit-transform property to a list of transform functions that includes the perspective(distance) function as the first transform. For example:

This text is rotated 45 degrees around its x-axis.

The foregoing snippet performs two 3D transforms on an element—rotation about the x-axis and perspective distortion, as if the user were viewing the object from 10 cm in front of the screen. Using 2D and 3D Transforms 3D Transforms 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 79Important: The perspective() transform function must be the first function in the list of transforms—you must establish the perspective prior to applying the other transforms. In almost all cases, it is better to create a 3D container and set the -webkit-perspective property for the container element than it isto apply the perspective() transform to an element directly. See Figure 7-7 (page 72) for details. Back Face Visibility If an element is rotated 90 degrees or more around the x- or y-axis, the back face of the element faces the user. The back face of an element is always transparent, so the user sees a reversed image of the front face through the transparent back face, like a sign painted on a glass door and seen from behind. To prevent the mirror image of the front face from being displayed, set the -webkit-backface-visibility property to hidden. For example: -webkit-backface-visibility: hidden; When -webkit-backface-visibility is set to hidden, an element is not displayed where its back face would be visible. One reason to do this is to create the illusion that an element has two faces, each with their own content. For example, to create the illusion of a card with different contents on the front and back face, two elements are positioned back to back in the same location. The two elements are then rotated together, progressively hiding the front element and revealing the back element. If the back face of the top element were visible, it would obscure the element beneath it instead of revealing the element beneath it as it rotates. Listing 7-8 creates the illusion of a card with content on both sides, as Figure 7-12 shows. Listing 7-8 Hiding the back side of a card flip card Using 2D and 3D Transforms Back Face Visibility 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 80
Back of card
Front of card
Figure 7-12 Cardflip example See the HTML version of this document to view the video. document to view the video. Using Transformation Matrices A transformation matrix is a small array of numbers (nine numbers for a 2D matrix, sixteen for a 3D matrix) used to transform another array, such as a bitmap, using linear algebra. Safari provides convenience functions for the most common matrix operations—translation, rotation, and scaling—but you can apply other transforms, such as reflection or shearing, by setting the matrix yourself. Using 2D and 3D Transforms Using Transformation Matrices 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 812D Matrix Operations For a 2D transform, set the -webkit-transform property to matrix(a,b,c,d,e,f), where the matrix position of the parametersisin column order, as Figure 7-13 shows. The first column in the matrix isthe x vector; the second column is the y vector. Figure 7-13 2D transformation matrix parameter positions c 0 0 1 b a d e f To make full use of transformation matrices, you need an understanding of linear algebra. But even without an understanding of linear algebra, you can often look up the matrix values for a particular effect. For example, here are the settings for reflection around the x- and y-axes: Reflection around the y-axis— -webkit-transform: matrix(-1,0,0,1,0,0); Reflection around the x-axis— -webkit-transform: matrix(1,0,0,-1,0,0); Here are the matrix parameter settings for some common effects: translate(x, y) = matrix(1, 0, 0, 1, x, y) scale(x, y) = matrix(x, 0, 0, y, 0, 0) rotate(a) = matrix(cos(a), sin(a), -sin(a), cos(a), 0, 0) skewx(a) = matrix(1, 0, tan(a), 1, 0, 0) skewy(a) = matrix(1, tan(a), 0, 1, 0, 0) An example of using matrix settingsto mirror,stretch, and skew elementsis given in Listing 7-9 and isillustrated in Figure 7-14. Listing 7-9 Matrix example Mirror transform
This element is reflected.
This element is reflected.
This element is reflected and stretched.
This element is reflected and stretched.
This element is reflected, stretched, and skewed.
This element is reflected, stretched, and skewed.
Figure 7-14 Matrix mirroring transforms Using 2D and 3D Transforms Using Transformation Matrices 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 833D Matrix Operations For a 3D transform, set the -webkit-transform property to matrix3d(a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p), where the parameters are a homogeneous 4 x 4 matrix in column-major order. This means that the a, b, c, and d parameters, for example, line up as the first column in a 4 x 4 matrix, as Figure 7-15 shows. The first column is the x vector, the second column is the y vector, and the third column is the z vector. Figure 7-15 3D matrix parameters Following are some common parameter settings for 3D transforms: ● Identity matrix—matrix3d(1,0,0,0,0,1,0,0,0,0,1,0,0,0,0,1) ● Translate matrix—matrix3d(1,0,0,tX,0,1,0,tY,0,0,1,tZ,0,0,0,1) ● Scale matrix—matrix3d(sX,0,0,0,0,sY,0,0,0,0,sZ,0,0,0,0,1) ● RotateX(a) matrix—matrix3d(1,0,0,0,0,cos(a),sin(a),0,0,sin(-a),cos(a),0,0,0,0,1) ● RotateY(a) matrix—matrix3d(cos(a),0,sin(-a),0,0,1,0,0,sin(a),0,cos(a),0,0,0,0,1) Working with Transforms in JavaScript There are a few things to be aware of when using JavaScript to control transforms. Using 2D and 3D Transforms Working with Transforms in JavaScript 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 84The CSS names for transform properties are different than the JavaScript names. When using JavaScript to set a property, delete the hyphens from the property name and capitalize the first letter of the following word. The following snippet shows how to set properties for an element with the ID "myElement" in CSS and JavaScript:

This page contains no images.

X Axis
Y Axis
Z Axis

Back

Front

Right

Left

Bottom

Using 2D and 3D Transforms Example: Animated Rotating Box Under JavaScript Control 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 88

Top


RotateX:
RotateY:

Figure 7-16 3D transforms under JavaScript control Using 2D and 3D Transforms Example: Animated Rotating Box Under JavaScript Control 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 89Use visual effects in combination with mouse and touch events to create interactive web applications that work equally well on the desktop and on iOS-based devices—applications that enable the users to manipulate objects on the screen with the mouse or with their fingers. Using a Click or Tap to Trigger a Transition Effect In this first example, the user can click an element using the mouse or tap an element using a finger to change the element’s size. The change in size is a 2D CSS transform animated using a CSS transition. A finger tap triggers an onclick event in Safari on iOS, so an event handler for onclick events is all you need to handle both tap and click events. First, the example in Listing 8-1 defines an class of element to manipulate and sets the properties needed to animate the transition using CSS. Next, the example defines a simple JavaScript function to change the element’s size using a CSS transform. Finally, the example installs the function as an event handler for onclick events on the element. Figure 8-1 shows the results. Listing 8-1 Simple touch or tap handler Click or tap to grow

Click or tap to grow.

Figure 8-1 Click and tap handler Controlling a 3D Transform with a Click, Touch, or Swipe In the next example, the user can flip a virtual page forward or backward with a click, a tap, or a swipe. Two div elements are styled to look like pages and are positioned back to back, and the example uses a 3D transform to create an animation that looks like the page of a book turning. Adding Interactive Control to Visual Effects Controlling a 3D Transform with a Click, Touch, or Swipe 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 91Any click, tap, or swipe on the page causes the page to turn, so in this case the direction of the swipe doesn’t matter. Both a single tap and a swipe gesture trigger a touchend event when they complete, so the example installs an event listener for the touchend event to handle taps and swipes as well as a listener for the click event. The example uses CSS to define the body element as a 3D container with perspective and to define a page class, with front and back subclasses. To make the page flip animation look like the page of a book turning, the div elements pivot on an edge, instead of around the middle. Consequently, the pages are positioned side by side,so that they lie on top of each other when one isfrontside-up and the other is backside-up, flipped 180 degrees around its edge. JavaScript functions flip the pages back and forth by applying 3D transforms. An onload function installs the page flipping functions as event listeners, with a different flip direction installed for the front and back subclasses. An additional listener function is installed for touchmove events to prevent a swipe from scrolling the browser window, which is the default behavior in Safari on iOS. There is a complication in using the touchend event to detect a tap while at the same time using the click event to detect a mouse click: a tap generates both a touchend event and a click event. To prevent the page flip from being called twice by a tap, an event listener is installed for the touchstart event. If a touchstart event occurs, the user is using a finger, not a mouse, so the listener functions for the click event are removed. Because the click event listeners need to be removed only once, the touchstart event listener removes itself as well. Adding Interactive Control to Visual Effects Controlling a 3D Transform with a Click, Touch, or Swipe 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 92The example uses HTML to create div elements of the page front and page back classes, to hold the page content, and to call the JavaScript function that installsthe event listeners on page load. The complete example is in Listing 8-2 and is illustrated in Figure 8-2. Figure 8-2 Page flip in action Listing 8-2 Page flip on click, tap, or swipe Page turner

Click, tap, or swipe to turn the page.

Back

Front

Using Gestures to Scale and Rotate Elements The final example uses pinch and rotate gestures to scale and rotate an element. Gesture events are high-level eventsthat encapsulate the lower-level touch events—they are instances of the GestureEvent class. Gesture and touch events can occur at the same time. Consequently, you can choose to handle touch events, gesture events, or both. The advantage of gesture events is that the location and angle of the fingers are properties of the events. Gesture events support pinching open and closed to zoom in and zoom out, and pivoting to rotate elements. Adding Interactive Control to Visual Effects Using Gestures to Scale and Rotate Elements 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 95The example in Listing 8-3 first defines the style of the element to manipulate using CSS. The example then declares three JavaScript functions: one function to set the element’s scale and rotation dynamically during a gesture; a second function to preserve the element’s current scale and rotation when the gesture ends; and a third function to install the first two functions as event listeners. Finally, the example uses HTML to create an element, give it content, and call the function that installsthe event listeners on page load. Figure 8-3 illustrates the result. Figure 8-3 Element rotated by a gesture Listing 8-3 Responding to gesture events pinch and rotate
Adding Interactive Control to Visual Effects Using Gestures to Scale and Rotate Elements 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 97

Pinch in to shrink.

Pinch out to grow.

Twirl to rotate.

Adding Interactive Control to Visual Effects Using Gestures to Scale and Rotate Elements 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 98This table describes the changes to Safari CSS Visual Effects Guide . Date Notes 2012-07-23 Documented support for CSS filters. 2012-01-13 Updated artwork and made minor edits. 2011-12-15 Edited and expanded. 2011-11-01 Edited for style and clarity. Updated, expanded, and revised for Safari 5.1 and iOS 5.0. Includes new examples and sample code. 2011-10-12 2011-07-20 Minor typo fix. 2010-11-03 Applied minor edits throughout. 2010-05-26 Made minor edits throughout. Added information on radial gradients and made minor changes to the chapter "Gradients." Added "CSS" to the title. 2010-02-24 2010-01-20 Minor edits. 2009-07-27 Minor edits throughout. 2009-06-08 Minor edits throughout. 2009-03-16 Added CSS gradients, masks, and reflections. New document that describes how to add visual effects to web content that is supported by Safari on the desktop and iOS. 2008-11-19 2012-07-23 | © 2012 Apple Inc. All Rights Reserved. 99 Document Revision HistoryApple Inc. © 2012 Apple Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrievalsystem, or transmitted, in any form or by any means, mechanical, electronic, photocopying, recording, or otherwise, without prior written permission of Apple Inc., with the following exceptions: Any person is hereby authorized to store documentation on a single computer for personal use only and to print copies of documentation for personal use provided that the documentation contains Apple’s copyright notice. No licenses, express or implied, are granted with respect to any of the technology described in this document. Apple retains all intellectual property rights associated with the technology described in this document. This document is intended to assist application developers to develop applications only for Apple-labeled computers. Apple Inc. 1 Infinite Loop Cupertino, CA 95014 408-996-1010 Apple, the Apple logo, Mac, Mac OS, OS X, Safari, and Spotlight are trademarks of Apple Inc., registered in the U.S. and other countries. Java is a registered trademark of Oracle and/or its affiliates. iOS is a trademark or registered trademark of Cisco in the U.S. and other countries and is used under license. Even though Apple has reviewed this document, APPLE MAKES NO WARRANTY OR REPRESENTATION, EITHER EXPRESS OR IMPLIED, WITH RESPECT TO THIS DOCUMENT, ITS QUALITY, ACCURACY, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.ASARESULT, THISDOCUMENT IS PROVIDED “AS IS,” AND YOU, THE READER, ARE ASSUMING THE ENTIRE RISK AS TO ITS QUALITY AND ACCURACY. IN NO EVENT WILL APPLE BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL,OR CONSEQUENTIAL DAMAGES RESULTING FROM ANY DEFECT OR INACCURACY IN THIS DOCUMENT, even if advised of the possibility of such damages. THE WARRANTY AND REMEDIES SET FORTH ABOVE ARE EXCLUSIVE AND IN LIEU OF ALL OTHERS, ORAL OR WRITTEN, EXPRESS OR IMPLIED. No Apple dealer, agent, or employee is authorized to make any modification, extension, or addition to this warranty. Some states do not allow the exclusion or limitation of implied warranties or liability for incidental or consequential damages, so the above limitation or exclusion may not apply to you. This warranty gives you specific legal rights, and you may also have other rights which vary from state to state. USB Device Interface GuideContents Introduction to USB Device Interface Guide 4 Organization of This Document 4 See Also 4 USB Device Overview 6 USB Device Types and Bus Speeds 6 USB Device Architecture and Terminology 7 USB Device Component Descriptors 8 USB Composite Class Devices 8 USB Transfer Types 8 Stalls and Halts 9 Data Synchronization in Non-Isochronous Transfers 10 USB 2.0 and Isochronous Transfers 10 USB Devices on OS X 11 Finding USB Devices and Interfaces 12 USB Family Error Codes 14 Determining Which Interface Version to Use 14 Tasks and Caveats 15 Handling Stalls, Halts, and Data Toggle Resynchronization 15 Using the Low Latency Isochronous Functions 15 Errors Reported by the EHCI Hub 17 Changes in Isochronous Functions to Support USB 2.0 17 USB Device Access in an Intel-Based Macintosh 18 Working With USB Device Interfaces 20 Using USB Device Interfaces 20 Accessing a USB Device 22 Definitions and Global Variables 22 The main Function 23 Working With the Raw Device 27 Working With the Bulk Test Device 34 Working With Interfaces 36 Document Revision History 46 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 2Tables and Listings USB Device Overview 6 Table 1-1 Examples of USB devices 6 Table 1-2 Keys for finding a USB device 12 Table 1-3 Keys for finding a USB interface 13 Working With USB Device Interfaces 20 Listing 2-1 Definitions and global variables 22 Listing 2-2 The main function 24 Listing 2-3 Accessing and programming the raw device 27 Listing 2-4 Releasing the raw device objects 30 Listing 2-5 Configuring a USB device 30 Listing 2-6 Two functions to download firmware to the raw device 32 Listing 2-7 Accessing the bulk test device 34 Listing 2-8 Finding interfaces on the bulk test device 36 Listing 2-9 Two asynchronous I/O completion functions 43 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 3Note: This document was previously titled Working With USB Device Interfaces. The I/O Kit provides a device interface mechanism that allows applications to communicate with and control hardware from outside the kernel. This document focuses on how to use that mechanism to create an application that detects the attachment of a USB device, communicates with it, and detects its detachment. This document does not describe how to develop an in-kernel driver for a USB modem or networking device. If you need to do this, refer to the documentation and sample code listed in “See Also” (page 4). Important: If your application is sandboxed, it must request the com.apple.security.device.usb entitlement in order to access USB devices. Organization of This Document This document contains the following chapters: ● “USB Device Overview” (page 6) provides an overview of USB device architecture and terminology and describes how USB devices are represented in OS X. ● “Working With USB Device Interfaces” (page 20) describes how to use the device interface mechanism to create a command-line tool that accesses a USB device. ● “Document Revision History” (page 46) lists the revisions of this document. See Also The ADC Reference Library contains several documents on device driver development for OS X and numerous sample drivers and applications. ● Accessing Hardware From Applications describes various ways to access devices from outside the kernel, including the device interface mechanism provided by the I/O Kit. For an overview of the I/O Kit terms and concepts used in this document, read the chapter Device Access and the I/O Kit. 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 4 Introduction to USB Device Interface Guide● I/O Kit Framework Reference contains API reference for I/O Kit methods and functions and for specific device families. ● Sample Code > Hardware & Drivers > USB includes both application-level and in-kernel code samples. Of particular relevance to this document is the application-level sample USBPrivateDataSample . ● OS X Man Pages provides access to existing reference documentation for BSD and POSIX functions and tools in a convenient HTML format. ● The usb mailing list provides a forum for discussing technical issues relating to USB devices in OS X. If you need to develop an in-kernel driver for a USB modem or networking device, refer to the following: ● I/O Kit Fundamentals describesthe architecture ofthe I/OKit,the object-oriented framework for developing OS X device drivers. ● ADC members can view the AppleUSBCDCDriver project in the source code for OS X v10.3.7 and later, available at Darwin Releases. To find the source code, select a version of OS X equal to or greater than v10.3.7 and click Source (choose the source for the PPC version, if there's a choice). This displays a new page, which lists the open source projects available for the version of OS X you've chosen. Scroll down to AppleUSBCDCDriver and click it to view the source. Be prepared to supply your ADC member name and password. ● Additional code samples that demonstrate specific in-kernel driver programming techniques are included as part of the OS X Developer Toolsinstallation package in /Developer/Examples/Kernel/IOKit/usb. If you're ready to create a universal binary version of your USB device-access application to run in an Intel-based Macintosh,seeUniversalBinaryProgrammingGuidelines.TheUniversalBinaryProgrammingGuidelines describes the differences between the Intel and PowerPC architectures and provides tips for developing a universal binary. If you are working with a device that complies with the USB mass storage specification but declares its device class to be vendor specific, see Mass Storage Device Driver Programming Guide for information on how to ensure the correct built-in driver loads for the device. Apple provides additional USB information (including the OS X USB Debug Kits) at http://developer.apple.com/hardwaredrivers/usb/index.html. A detailed description of the USB device specification is beyond the scope of this document—for more information, see Universal Serial Bus Specification Revision 2.0 available at http://www.usb.org. Introduction to USB Device Interface Guide See Also 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 5This chapter provides a summary of USB device architecture and describes how USB devices are represented in OS X. It also presents a few specific guidelines for working with USB devices in an application.For details on the USB specification, see http://www.usb.org. USB Device Types and Bus Speeds The USB specification supports a wide selection of devices that range from lower-speed devices such as keyboards, mice, and joysticks to higher-speed devices such as scanners and digital cameras. The specification lists a number of device classes that each define a set of expected device behaviors. Table 1-1 (page 6) lists some examples of USB devices, categorized by class. Table 1-1 Examples of USB devices USB device class USB devices in class Audio class Speakers, microphones Chip Card Interface Device Class Smart cards, chip cards Communication class Speakerphone, modem A device in which all class-specific information is embedded in its interfaces Composite class HID class Keyboards, mice, joysticks, drawing tablets Hub class Hubs provide additional attachment points for USB devices Hard drives, flash memory readers, CD Read/Write drives, digital cameras, and high-end media players Mass storage class Printing class Printers A device that doesn’t fit into any other predefined class or one that doesn’t use the standard protocols for an existing class Vendor specific Digital camcorders, webcams, digital still cameras that support video streaming Video class 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 6 USB Device OverviewVersion 1.1 of the USB specification supports two bus speeds: ● Low speed (1.5 Mbps) ● Full speed (12 Mbps) Version 2.0 of the specification adds another bus speed to this list: ● High speed (480 Mbps) The USB 2.0 specification is fully compatible with low-speed and full-speed USB devices and even supports the use of cables and connectors made to meet earlier versions of the specification. Apple provides USB 2.0 ports on all new Macintosh computers and fully supports the new specification with Enhanced Host Controller Interface (EHCI) controllers and built-in, low-level USB drivers. For the most part, you do not have to change existing applications to support the faster data rate because the speed increase and other enhancements are implemented at such a low level. The exceptions to this are some differences in isochronous transfers. For information on how the USB 2.0 specification affects isochronous transfers, see “USB 2.0 and Isochronous Transfers” (page 10). USB Device Architecture and Terminology The architecture of a generic USB device is multi-layered. A device consists of one or more configurations, each of which describes a possible setting the device can be programmed into. Such settings can include the power characteristics of the configuration (for example, the maximum power consumed by the configuration and whether it is self-powered or not) and whether the configuration supports remote wake-up. Each configuration contains one or more interfacesthat are accessible after the configuration isset. An interface provides the definitions of the functions available within the device and may even contain alternate settings within a single interface. For example, an interface for an audio device may have different settings you can select for different bandwidths. Each interface contains zero or more endpoints. An endpoint is a uniquely identifiable portion of a USB device that is the source or sink of information in a communication flow between the host and the device. Each endpoint has characteristics that describe the communication it supports, such as transfer type (control, isochronous, interrupt, or bulk, described in “USB Transfer Types” (page 8)), maximum packet size, and transfer direction (input or output). Communication with a USB device is accomplished through a pipe, a logical association between an endpoint and software running on the host. Endpoint and pipe are often used synonymously although an endpoint is a component of a USB device and a pipe is a logical abstraction of the communications link between endpoint and host. USB Device Overview USB Device Architecture and Terminology 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 7USB Device Component Descriptors Each layer of a USB device providesinformation about its attributes and resource requirementsin its descriptor, a data structure accessible through device interface functions. By examining the descriptors at each layer, you can determine exactly which endpoint you need to communicate successfully with a particular device. At the top layer is the device descriptor, which has fields associated with information such as the device’s class and subclass, vendor and product numbers, and number of configurations. Each configuration in turn has a configuration descriptor containing fields that describe the number of interfaces it supports and the power characteristics of the device when it is in that configuration, along with other information. Each interface supported by a configuration has its own descriptor with fields for information such as the interface class, subclass, and protocol, and the number of endpoints in that interface. At the bottom layer are the endpoint descriptors that specify attributes such as transfer type and maximum packet size. The USB specification defines a name for each descriptor field, such as the bDeviceClass field in the device descriptor and the bNumInterfaces field in the configuration descriptor, and each field is associated with a value. For a complete listing of all descriptor fields, see the USB specification at www.usb.org. The USB family defines structures that represent the descriptors defined by the USB specification. For the definitions of these structures, see USB in Kernel Framework Reference . USB Composite Class Devices The USB specification defines a composite class device as a device whose device-descriptor fields for device class (bDeviceClass) and device subclass (bDeviceSubClass) both have the value 0. A composite class device appears to the system as a USB device using a single bus address that may present multiple interfaces, each of which represents a separate function. A good example of a composite class device is a multifunction device, such as a device that performs printing, scanning, and faxing. In such a device, each function is represented by a separate interface. In OS X, the I/O Kit loads the AppleUSBComposite device driver for composite class devices that do not already have vendor-specific device drivers to drive them. The AppleUSBComposite driver configures the device and causes drivers to be loaded for each USB interface. Although most multifunction USB devices are composite class devices, not all composite class devices are multifunction devices. The manufacturer of a single-function USB device is at liberty to classify the device as a composite class device as long as the device meets the USB specifications. For more information on how OS X represents USB devices and interfaces, see “USB Devices on OS X” (page 11). USB Transfer Types The USB specification defines four types of pipe transfer: ● Control—intended to support configuration, command, and status communication between the host software and the device. Control transfers support error detection and retry. USB Device Overview USB Device Architecture and Terminology 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 8● Interrupt—used to support small, limited-latency transfers to or from a device such as coordinates from a pointing device or status changes from a modem. Interrupt transfers support error detection and retry. ● Isochronous—used for periodic, continuous communication between the host and the device, usually involving time-relevant information such as audio or video data streams. Isochronous transfers do not support error detection or retry. ● Bulk—intended for non-periodic, large-packet communication with relaxed timing constraints such as between the host software and a printer or scanner. Bulk transfers support error detection and retry. Pipes also have a transfer direction associated with them. A control pipe can support bidirectional communication but all other pipes are strictly uni-directional. Therefore, two-way communication requires two pipes, one for input and one for output. Every USB device is required to implement a default control pipe that provides access to the device’s configuration, status, and control information. This pipe, implemented in the IOUSBDevice nub object (described in “USB Devices on OS X” (page 11)), is used when a driver such as the AppleUSBComposite driver configures the device or when device-specific control and status information is needed. For example, your application would use the default control pipe if it needs to set or choose a configuration for the device. The default control pipe is connected to the default endpoint (endpoint 0). Note that endpoint 0 does not provide an endpoint descriptor and it is never counted in the total number of endpoints in an interface. The interfaces associated with a configuration can contain any combination of the three remaining pipe types (interrupt, isochronous, and bulk), implemented in the IOUSBInterface nub objects (described in “USB Devices on OS X” (page 11)). Your application can query the interface descriptors of a device to select the pipe most suited to its needs. Stalls and Halts Although a stall and a halt are different, they are closely related in their effect on data transmission. Halt is a feature of an endpoint and it can be set by either the host or the device itself in response to an error. A stall is a type of handshake packet an endpoint returns when it is unable to transmit or receive data or when its halt feature is set (the host never sends a stall packet). When an endpoint sends a stall packet, the host can halt the endpoint. Depending on the precise circumstances and on how compliant the device is, the halt feature must be cleared in the host, the endpoint, or both before data transmission can resume. When the halt is cleared the data toggle bit, used to synchronize data transmission, is also reset (see “Data Synchronization in Non-Isochronous Transfers” (page 10) for more information about the data toggle). For information on how to handle these conditions in your application, see “Handling Stalls, Halts, and Data Toggle Resynchronization” (page 15). USB Device Overview USB Device Architecture and Terminology 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 9Data Synchronization in Non-Isochronous Transfers The USB specification defines a simple protocol to provide data synchronization across multiple packets for non-isochronoustransfers(recall that isochronoustransfers do notsupport error recovery or retry). The protocol is implemented by means of a data toggle bit in both the host and the endpoint which is synchronized at the start of a transaction (or when a reset occurs). The precise synchronization mechanism varies with the type of transfer; see the USB specification for details. Both the host and the endpoint begin a transaction with their data toggle bitsset to zero. In general, the entity receiving data toggles its data toggle bit when it is able to accept the data and it receives an error-free data packet with the correct identification. The entity sending the data toggles its data toggle bit when it receives a positive acknowledgement from the receiver. In this way, the data toggle bits stay synchronized until, for example, a packet with an incorrect identification is received. When this happens, the receiver ignores the packet and does not increment its data toggle bit. When the data toggle bits get out of synchronization (for this or any other reason), you will probably notice that alternate transactions are not getting through in your application. The solution to this is to resynchronize the data toggle bits. For information on how to do this, see “Handling Stalls, Halts, and Data Toggle Resynchronization” (page 15). USB 2.0 and Isochronous Transfers The USB 2.0 specification supports the same four transfer types as earlier versions of the specification. In addition to supporting a higher transfer rate, the new specification defines an improved protocol for high-speed transfers and new ways of handling transactions for low-speed and full-speed devices. For details on the protocols and transaction-handling methods, see the specification at http://www.usb.org. For the most part, these enhancements are implemented at the hostsoftware level and do not require changes to your code. For isochronous transfers, however, you should be aware of the following differences: ● Earlier versions of the specification divide bus time into 1-millisecond frames, each of which can carry multiple transactionsto multiple destinations. (A transaction containstwo or more packets: a token packet and one or more data packets, a handshake packet, or both.) The USB 2.0 specification divides the 1-millisecond frame into eight, 125-microsecond microframes, each of which can carry multiple transactions to multiple destinations. ● The maximum amount of data allowed in a transaction is increased to 3 KB. ● Any isochronous endpoints in a device’s default interface must have a maximum packet size of zero. (This means that the default setting for an interface containing isochronous pipes is alternate setting zero and the maximum packet size for that interface’s isochronous endpoints must be zero.) This ensures that the host can configure the device no matter how busy the bus is. For a summary of how these differences affect the OS X USB API, see “Changes in Isochronous Functions to Support USB 2.0” (page 17). USB Device Overview USB Device Architecture and Terminology 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 10USB Devices on OS X When a USB device is plugged in, the OS X USB family abstracts the contents of the device descriptor into an I/O Kit nub object called an IOUSBDevice. This nub object is attached to the IOService plane of the I/O Registry as a child of the driver for the USB controller. The IOUSBDevice nub object is then registered for matching with the I/O Kit. If the device is a composite class device with no vendor-specific driver to match against it, the AppleUSBComposite driver matches against it and starts as its provider. The AppleUSBComposite driver then configures the device by setting the configuration in the device’s list of configuration descriptors with the maximum power usage that can be satisfied by the port to which the device is attached. This allows a device with a low power and a high power configuration to be configured differently depending on whether it’s attached to a bus-powered hub or a self-powered hub. In addition, if the IOUSBDevice nub object has the “Preferred Configuration” property, the AppleUSBComposite driver will always use that value when it attempts to configure the device. The configuration of the device causes the USB family to abstract each interface descriptor in the chosen configuration into an IOUSBInterface nub object. These nub objects are attached to the I/O Registry as children of the original IOUSBDevice nub object and are registered for matching with the I/O Kit. Important: Because a composite class device is configured by the AppleUSBComposite driver, setting the configuration again from your application will result in the destruction of the IOUSBInterface nub objects and the creation of new ones. In general, the only reason to set the configuration of a composite class device that’s matched by the AppleUSBComposite driver is to choose a configuration other than the first one. For non-composite class devices or composite class devices with vendor-specific drivers that match against them, there is no guarantee that any configuration will be set and you may have to perform this task within your application. It's important to be mindful of the difference between a USB device (represented in the I/O Registry by an IOUSBDevice nub object) and its interfaces (each represented by an IOUSBInterface nub object). A multifunction USB device, for example, is represented in the I/O Registry by one IOUSBDevice object and one IOUSBInterface object for each interface. The distinction between interface and device isimportant because it determines which object your application must find in the I/O Registry and which type of device interface to get. For example, if your application needs to communicate with a specific interface in a multifunction USB device, it must find that interface and get an IOUSBInterfaceInterface to communicate with it. An application that needs to communicate with the USB device as a whole, on the other hand, would need to find the device in the I/O Registry and get an IOUSBDeviceInterface to communicate with it. For more information on finding devices and interfaces in USB Device Overview USB Devices on OS X 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 11the I/O Registry, see “Finding USB Devices and Interfaces” (page 12); for more information on how to get the proper device interface to communicate with a device or interface, see “Using USB Device Interfaces” (page 20). Finding USB Devices and Interfaces To find a USB device or interface, use the keys defined in the Universal Serial Bus Common Class Specification, Revision 1.0 (available for download from http://www.usb.org/developers/devclass_docs/usbccs10.pdf) to create a matching dictionary that defines a particular search. If you are unfamiliar with the concept of device matching, see the section “Finding Devices in the I/O Registry” in Accessing Hardware From Applications. The keys defined in the specification are listed in the tables below. Each key consists of a specific combination of elements in a device or interface descriptor. In the tables below, the elements in a key are separated by the ‘+’ character to emphasize the requirement that all a key’s elements must appear together in your matching dictionary. Both tables present the keys in order of specificity: the first key in each table defines the most specific search and the last key defines the broadest search. Before you build a matching dictionary, be sure you know whether your application needs to communicate with a device or a specific interface in a device. It’s especially important to be aware of this distinction when working with multifunction devices. A multifunction device is often a composite class device that defines a separate interface for each function. If, for example, your application needs to communicate with the scanning function of a device that does scanning, faxing, and printing, you need to build a dictionary to match on only the scanning interface (an IOUSBInterface object), not the device as a whole (an IOUSBDevice object). In this situation, you would use the keys defined for interface matching (those shown in Table 1-3 (page 13)), not the keys for device matching. Table 1-2 (page 12) lists the keys you can use to find devices (not interfaces). Each key element is a piece of information contained in the device descriptor for a USB device. Table 1-2 Keys for finding a USB device Key Notes bcdDevice contains the release number of the device idVendor + idProduct + bcdDevice idVendor + idProduct Use this key only if the device’s bDeviceClass is $FF idVendor + bDeviceSubClass + bDeviceProtocol Use this key only if the device’s bDeviceClass is $FF idVendor + bDeviceSubClass USB Device Overview USB Devices on OS X 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 12Key Notes Use this key only if the device’s bDeviceClass is not $FF bDeviceClass + bDeviceSubClass + bDeviceProtocol Use this key only if the device’s bDeviceClass is not $FF bDeviceClass + bDeviceSubClass Table 1-3 (page 13) lists the keys you can use to find interfaces (not devices). Each key element is a piece of information contained in an interface descriptor for a USB device. Table 1-3 Keys for finding a USB interface Key Notes idVendor + idProduct + bcdDevice + bConfigurationValue + bInterfaceNumber idVendor + idProduct + bConfigurationValue + bInterfaceNumber Use this key only if bInterfaceClass is $FF idVendor + bInterfaceSubClass + bInterfaceProtocol Use this key only if bInterfaceSubClass is $FF idVendor + bInterfaceSubClass Use this key only if bInterfaceSubClass is not $FF bInterfaceClass + bInterfaceSubClass + bInterfaceProtocol Use this key only if bInterfaceSubClass is not $FF bInterfaceClass + bInterfaceSubClass For a successful search, you must add the elements of exactly one key to your matching dictionary. If your matching dictionary contains a combination of elements not defined by any key, the search will be unsuccessful. For example, if you create a matching dictionary containing values representing a device’s vendor, product, and protocol, the search will be unsuccessful even if a device with those precise values in its device descriptor is currently represented by an IOUSBDevice nub in the I/O Registry. This is because there is no key in Table 1-2 (page 12) that combines the idVendor, idProduct, and bDeviceProtocol elements. USB Device Overview USB Devices on OS X 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 13USB Family Error Codes As you develop an application to access a USB device or interface, you will probably encounter error codes specific to the OS X USB family. If you are using Xcode, you can search for information about these error codes in the Xcode documentation window. To find error code documentation, select Documentation from the Xcode Help menu. Select Full-Text Search from the pull-down menu associated with the search field (click the magnifying glass icon to reveal the menu). Select Reference Library in the Search Groups pane at the left of the window. Type an error code number in the search field, such as 0xe0004057, and press Return. Select the most relevant entry in the search results to display the document in the lower portion of the window. Use the Find command (press Command-F) to find the error code in this document. Using the example of error code 0xe0004057, you’ll see that this error is returned when the endpoint has not been found. For help with deciphering I/O Kit error codes in general, see Technical Q&A QA1075, “Making sense of I/O Kit error codes.” Determining Which Interface Version to Use As described in “USB Devices on OS X” (page 11), the OS X USB family provides an IOUSBDeviceInterface object you use to communicate with a USB device as a whole and an IOUSBInterfaceInterface object you use to communicate with an interface in a USB device. There are a number of different versions of the USB family, however, some of which provide new versions of these interface objects. (One way to find the version of the USB family installed in your computer is to view the Finder preview information for the IOUSBFamily.kext located in /System/Library/Extensions.) This section describes how to make sure you use the correct interface object and how to view the documentation for the interface objects. The first version of the USB family was introduced in OS X v10.0 and contains the first versions of the interface objects IOUSBDeviceInterface and IOUSBInterfaceInterface. When new versions of the USB family introduce new functions for an interface object, a new version of the interface object is created, which gives access to both the new functions and all functions defined in all previous versions of that interface object. For example, the IOUSBDeviceInterface197 object provides two new functions you can use with version 1.9.7 of the USB family (available in OS X v10.2.3 and later), in addition to all functions available in the previous device interface objects IOUSBDeviceInterface187, IOUSBDeviceInterface182, and IOUSBDeviceInterface. As you develop an application that accesses a USB device or interface, you should use the latest version of the interface object that is available in the earliest version of OS X that you want to support. For example, if your application must run in OS X v10.0, you must use the IOUSBDeviceInterface and IOUSBInterfaceInterface objects. If, however, you develop an application to run in OS X v10.4 and later, you use the IOUSBDeviceInterface197 object to access the device as a whole and the USB Device Overview USB Devices on OS X 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 14IOUSBInterfaceInterface220 object to access an interface in it. This is because IOUSBDeviceInterface197 is available inOS X version 10.2.3 and later and IOUSBInterfaceInterface220 is available in OS X v10.4 and later. Note: When you view the documentation for these interface objects, notice that each version is documented separately. For example, the documentation for IOUSBDeviceInterface197 contains information about the two new functions introduced in this version, but does not repeat the documentation for the functions introduced in IOUSBDeviceInterface187, IOUSBDeviceInterface182, and IOUSBDeviceInterface. Tasks and Caveats This section presents some specific tasks your application might need to perform, along with some caveats related to USB 2.0 support of which you should be aware. Handling Stalls, Halts, and Data Toggle Resynchronization As described in “Stalls and Halts ” (page 9), stalls and halts are closely related in their effect on data transmission. To simplify the API, the USB family uses the pipe stall terminology in the names of the functions that handle these conditions: ● ClearPipeStall ● ClearPipeStallBothEnds The ClearPipeStall function operates exclusively on the host controller side, clearing the halt feature and resetting the data toggle bit to zero. If the endpoint’s halt feature and data toggle bit must be reset as well, your application must do so explicitly, using one of the ControlRequest functions to send the appropriate device request. See the documentation for the USB.h header file in I/O Kit Framework Reference for more information about standard device requests. In OS X version 10.2 and later, you can use the ClearPipeStallBothEnds function which, as its name suggests, clears the halt and resets the data toggle bit on both sides at the same time. Using the Low Latency Isochronous Functions In OS X, the time between when an isochronous transaction completes on the USB bus and when you receive your callback can stretch to tens of milliseconds. This is because the callback happens on the USB family work loop, which runs at a lower priority than some other threads in the system. In most cases, you can work around USB Device Overview Tasks and Caveats 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 15this delay by queuing read and write requests so that the next transaction is scheduled and ready to start before you receive the callback from the current transaction. In fact, this scheme is a good way to achieve higher performance whether or not low latency is a requirement of your application. In a few cases, however, queuing isochronous transactions to keep the pipe busy is not enough to prevent a latency problem that a user might notice. Consider an application that performs audio processing on some USB input (from a musical instrument, for example) before sending the processed data out to USB speakers. In this scenario, a user hears both the raw, unprocessed output of the instrument and the processed output of the speakers. Of course, some small delay between the time the instrument creates the raw sound waves and the time the speaker emits the processed sound waves is unavoidable. If this delay is greater than about 8 milliseconds, however, the user will notice. In OS X version 10.2.3 (version 1.9.2 of the USB family) the USB family solves this problem by taking advantage of the predictability of isochronous data transfers. By definition, isochronous mode guarantees the delivery of some amount of data every frame or microframe. In earlier versions of OS X, however, it was not possible to find out the exact amount of data that was transferred by a given time. This meant that an application could not begin processing the data until it received the callback associated with the transaction, telling it the transfer status and the actual amount of data that was transferred. Version 1.9.2 of the USB family introduced the LowLatencyReadIsochPipeAsync and LowLatencyWriteIsochPipeAsync functions. These functions update the frame list information (including the transferstatus and the number of bytes actually transferred) at primary interrupt time. Using these functions, an application can request that the frame list information be updated as frequently as every millisecond. This means an application can retrieve and begin processing the number of bytes actually transferred once a millisecond, without waiting for the entire transaction to complete. Important: Because these functions cause processing at primary interrupt time, it is essential you use them only if it is absolutely necessary. Overuse of these functions can cause degradation of system performance. To support the low latency isochronous read and write functions, the USB family also introduced functions to create and destroy the buffers that hold the frame list information and the data. Although you can choose to create a single data buffer and a single frame list buffer or multiple buffers of each type, you must use the LowLatencyCreateBuffer function to create them. Similarly, youmust use the LowLatencyDestroyBuffer function to destroy the buffers after you are finished with them. This restricts all necessary communication with kernel entities to the USB family. For reference documentation on the low latency isochronous functions, see the IOUSBLib.h documentation in I/O Kit Framework Reference . USB Device Overview Tasks and Caveats 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 16Errors Reported by the EHCI Hub The EHCI hub that supports high-speed devices (as well as low-speed and full-speed devices) provides coarser-grained error reporting than the OHCI hub does. For example, with an OHCI hub, you might receive an “endpoint timed out” error if you unplug the device while it is active. If you perform the same action with an EHCI hub, you might receive a “pipe stalled” error instead. The Apple EHCI hub driver cannot get more detailed error information from the hub, so it alternates between reporting “device not responding” and “pipe stalled” regardless of the actual error reported by the device. To avoid problems with your code, be sure your application does not rely on other, more specific errors to make important decisions. Changes in Isochronous Functions to Support USB 2.0 Recall that the USB 2.0 specification divides the 1-millisecond frame into eight, 125-microsecond microframes. The USB family handles this by reinterpreting some function parameters (where appropriate) and adding a couple of new functions. This section summarizes these changes; for reference documentation, see documentation for IOUSBLib.h in I/O Kit Framework Reference . The functions you use to read from and write to isochronous endpoints are ReadIsochPipeAsync and WriteIsochPipeAsync. Both functions include the following two parameters: ● numFrames—The number of frames for which to transfer data ● frameList—A pointer to an array of structures that describe the frames If you need to handle high-speed isochronous transfers, you can think of these parameters as referring to “transfer opportunities” instead of frames. In other words, numFrames can refer to a number of frames for full-speed devices or to a number of microframes for high-speed devices. Similarly, frameList specifies the list of transfers you want to occur, whether they are in terms of frames or microframes. Note: The ReadIsochPipeAsync and WriteIsochPipeAsync functions also have the frameStart parameter in common, but it does not get reinterpreted. Thisis because all isochronoustransactions, including high-speed isochronoustransactions,start on a frame boundary, not amicroframe boundary. To help you determine whether a device isfunctioning in full-speed or high-speed mode, the USB family added the GetFrameListTime function, which returns the number of microseconds in a frame. By examining the result (kUSBFullSpeedMicrosecondsInFrame or kUSBHighSpeedMicrosecondsInFrame) you can tell in which mode the device is operating. USB Device Overview Tasks and Caveats 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 17The USB family also added the GetBusMicroFrameNumber function which is similar to the GetBusFrameNumber function, except that it returns both the current frame and microframe number and includes the time at which that information was retrieved. To handle the new specification’s requirement that isochronous endpoints in a device’s default interface have a maximum packetsize of zero, the USB family added functionsthat allow you to balance bandwidth allocations among isochronous endpoints. A typical scenario is this: 1. Call GetBandwidthAvailable (available inOS X version 10.2 and later)to determine howmuch bandwidth is currently available for allocation to isochronous endpoints. 2. Call GetEndpointProperties (available in OS X version 10.2 and later) to examine the alternate settings of an interface and find one that uses an appropriate amount of bandwidth. 3. Call SetAlternateInterface (available in OS X version 10.0 and later) to create the desired interface and allocate the pipe objects. 4. Call GetPipeProperties (available in OS X version 10.0 and later) on the chosen isochronous endpoint. Thisis a very importantstep because SetAlternateInterface willsucceed, even if there is not enough bandwidth for the endpoints. Also, another device might have claimed the bandwidth that was available at the time the GetBandwidthAvailable function returned. If this happens, the maximum packet size for your chosen endpoint (contained in the maxPacketSize field) is now zero, which means that the bandwidth is no longer available. In addition, in OS X version 10.2, the USB family added the SetPipePolicy function, which allows you to relinquish bandwidth that might have been specified in an alternate setting. USB Device Access in an Intel-Based Macintosh This section provides an overview of some of the issues related to developing a universal binary version of an application that accesses a USB device. Before you read this section, be sure to read Universal Binary Programming Guidelines. That document covers architectural differences and byte-ordering formats and provides comprehensive guidelines for code modification and building universal binaries. The guidelines in that document apply to all types of applications, including those that access hardware. Before you build your application as a universal binary, make sure that: ● You port your project to GCC 4 (Xcode uses GCC 4 to target Intel-based Macintosh computers) ● You install the OS X v10.4 universal SDK ● You develop your project in Xcode 2.1 or later USB Device Overview USB Device Access in an Intel-Based Macintosh 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 18The USB bus is a little-endian bus. Structured data appears on the bus in the little-endian format regardless of the native endian format of the computer an application isrunning in. If you've developed a USB device-access application to run in a PowerPC-based Macintosh, you probably perform some byte swapping on data you read from the USB bus because the PowerPC processor uses the big-endian format. For example, the USB configuration descriptor structure contains a two-byte field that holds the descriptor length. If your PowerPC application reads this structure from the USB bus (instead of receiving it from a USB device interface function), you need to swap the value from the USB bus format (little endian) to the PowerPC format (big endian). The USB family provides several swapping macros that swap from USB to host and from host to USB (for more information on these macros, see USB.h). The Kernel framework also provides byte-swapping macros and functions you can use in high-level applications (see the OSByteOrder.h header file in libkern). If you use these macros in your application, you shouldn't have any trouble developing a universal binary version of your application. This is because these macros determine at compile time if a swap is necessary. If, however, your application uses hard-coded swaps from little endian to big endian, your application will not run correctly in an Intel-based Macintosh. As you develop a universal binary version of your application, therefore, be sure to use the USB family swapping macros or the macros in libkern/OSByteOrder.h for all byte swapping. Although you may need to perform byte swapping on values your application reads from the USB bus, you do not need to perform any byte swapping on values you pass in arguments to functions in the USB family API. You should pass argument values in the computer's host format. Likewise, any values you receive from the USB family functions will be in the computer's host format. USB Device Overview USB Device Access in an Intel-Based Macintosh 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 19This chapter describes how to develop a user-space tool that finds and communicates with an attached USB device and one of its interfaces. Important: The sample code featured in this document isintended to illustrate how to access a USB device from an application. It is not intended to provide guidance on error handling and other features required for production-quality code. Using USB Device Interfaces Applications running in OS X get access to USB devices by using I/O Kit functions to acquire a device interface, a type of plug-in that specifies functions the application can call to communicate with the device. The USB family provides two types of device interface: ● IOUSBDeviceInterface for communicating with the device itself ● IOUSBInterfaceInterface for communicating with an interface in the device Both device interfaces are defined in /System/Library/Frameworks/IOKit.framework/Headers/usb/IOUSBLib.h. Communicating with the device itself is usually only necessary when you need to set or change its configuration. For example, vendor-specific devices are often not configured because there are no default drivers that set a particular configuration. In this case, your application must use the device interface for the device to set the configuration it needs so the interfaces become available. Important: If your application is sandboxed, it must request the com.apple.security.device.usb entitlement in order to access USB devices. The process of finding and communicating with a USB device is divided into two sets of steps. The first set outlines how to find a USB device, acquire a device interface of type IOUSBDeviceInterface for it, and set or change its configuration. The second set describes how to find an interface in a device, acquire a device interface of type IOUSBInterfaceInterface for it, and use it to communicate with that interface. If you need to communicate with an unconfigured device or if you need to change a device’s configuration, you follow both sets of steps. If you need to communicate with a device that is already configured to your 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 20 Working With USB Device Interfacesspecification, you follow only the second set of steps. The sample code in “Accessing a USB Device” (page 22) follows both sets of steps and extends them to include setting up notifications it can receive when devices are dynamically added or removed. Follow this first set of steps only to set or change the configuration of a device. If the device you’re interested in is already configured for your needs, skip these steps and follow the second set of steps. 1. Find the IOUSBDevice object that represents the device in the I/O Registry. This includes setting up a matching dictionary with a key from the USB Common Class Specification (see “Finding USB Devices and Interfaces” (page 12)). The sample code usesthe key elements kUSBVendorName and kUSBProductName to find a particular USB device (this is the second key listed in Table 1-2 (page 12)). 2. Create a device interface of type IOUSBDeviceInterface for the device. This device interface provides functionsthat perform taskssuch assetting or changing the configuration of the device, getting information about the device, and resetting the device. 3. Examine the device’s configurations with GetConfigurationDescriptorPtr, choose the appropriate one, and call SetConfiguration to set the device’s configuration and instantiate the IOUSBInterface objects for that configuration. Follow thissecond set ofstepsto find and choose an interface, acquire a device interface for it, and communicate with the device. 1. Create an interface iterator to iterate over the available interfaces. 2. Create a device interface for each interface so you can examine its properties and select the appropriate one. To do this, you create a device interface of type IOUSBInterfaceInterface. This device interface providesfunctionsthat perform taskssuch as getting information about the interface,setting the interface’s alternate setting, and accessing its pipes. 3. Use the USBInterfaceOpen function to open the selected interface. This will cause the pipes associated with the interface to be instantiated so you can examine the properties of each and select the appropriate one. 4. Communicate with the device through the selected pipe. You can write to and read from the pipe synchronously or asynchronously—the sample code in “Accessing a USB Device” (page 22) shows how to do both. Working With USB Device Interfaces Using USB Device Interfaces 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 21Accessing a USB Device This section provides snippets of sample code that show how to access a Cypress EZ-USB chip with an 8051 microcontroller core. The sample code followsthe firstset ofstepsin section “Using USB Device Interfaces” (page 20) to find the Cypress EZ-USB chip in its default, unprogrammed state (also referred to as the “raw device”). It then configures the device and downloads firmware provided by Cypress to program the chip to behave as a device that echoes all information it receives on its bulk out pipe to its bulk in pipe. Once the chip has been programmed, the device nub representing the default, unprogrammed device is detached from the I/O Registry and a new device nub, representing the programmed chip, is attached. To communicate with the programmed chip (also referred to as the “bulk test device”), the sample code must perform the first set of steps again to find the device, create a device interface for it, and configure it. Then it performs the second set of steps to find an interface, create a device interface for it, and test the device. The sample code also shows how to set up notifications for the dynamic addition and removal of a device. Important: If your application is sandboxed, it must request the com.apple.security.device.usb entitlement in order to access USB devices. Definitions and Global Variables The code in the USB Notification Example uses the definitions and global variables shown in Listing 2-1 (page 22). The definition of USE_ASYNC_IO allows you to choose to use either synchronous or asynchronous calls to read from and write to the chip by commenting out the line or leaving it in, respectively. The definition of kTestMessage sets up a simple message to write to the device. The remaining definitions are specific to the Cypress EZ-USB chip. Listing 2-1 Definitions and global variables #define USE_ASYNC_IO //Comment this line out if you want to use //synchronous calls for reads and writes #define kTestMessage "Bulk I/O Test" #define k8051_USBCS 0x7f92 #define kOurVendorID 1351 //Vendor ID of the USB device #define kOurProductID 8193 //Product ID of device BEFORE it //is programmed (raw device) #define kOurProductIDBulkTest 4098 //Product ID of device AFTER it is //programmed (bulk test device) //Global variables Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 22static IONotificationPortRef gNotifyPort; static io_iterator_t gRawAddedIter; static io_iterator_t gRawRemovedIter; static io_iterator_t gBulkTestAddedIter; static io_iterator_t gBulkTestRemovedIter; static char gBuffer[64]; The main Function The main function in the USB Notification Example project (contained in the file main.c) accomplishes the following tasks. ● It establishes communication with the I/O Kit and sets up a matching dictionary to find the Cypress EZ-USB chip. ● It sets up an asynchronous notification to be called when an unprogrammed (raw) device is first attached to the I/O Registry and another to be called when the device is removed. ● It modifies the matching dictionary to find the programmed (bulk test) device. ● It sets up additional notifications to be called when the bulk test device is first attached or removed. ● It starts the run loop so the notifications that have been set up will be received. The main function uses I/O Kit functions to set up and modify a matching dictionary and set up notifications, and Core Foundation functions to set up the run loop for receiving the notifications. It calls the following functions to access both the raw device and the bulk test device. ● RawDeviceAdded, shown in Listing 2-3 (page 27), iterates over the set of matching devices and creates a device interface for each one. It calls ConfigureDevice (shown in Listing 2-5 (page 30)) to set the device’s configuration, and then DownloadToDevice (shown in Listing 2-6 (page 32)) to download the firmware to program it. ● RawDeviceRemoved,shown in Listing 2-4 (page 30), iterates over the set of matching devices and releases each one in turn. ● BulkTestDeviceAdded, shown in Listing 2-7 (page 34), iterates over the new set of matching devices, creates a device interface for each one, and calls ConfigureDevice (shown in Listing 2-5 (page 30)) to set the device’s configuration. It then calls FindInterfaces (shown in Listing 2-8 (page 36)) to get access to the interfaces on the device. ● BulkTestDeviceRemoved iterates over the new set of matching devices and releases each one in turn. This function is not shown in this chapter; see RawDeviceRemoved (Listing 2-4 (page 30)) for a nearly identical function. Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 23Listing 2-2 The main function int main (int argc, const char *argv[]) { mach_port_t masterPort; CFMutableDictionaryRef matchingDict; CFRunLoopSourceRef runLoopSource; kern_return_t kr; SInt32 usbVendor = kOurVendorID; SInt32 usbProduct = kOurProductID; // Get command line arguments, if any if (argc > 1) usbVendor = atoi(argv[1]); if (argc > 2) usbProduct = atoi(argv[2]); //Create a master port for communication with the I/O Kit kr = IOMasterPort(MACH_PORT_NULL, &masterPort); if (kr || !masterPort) { printf("ERR: Couldn’t create a master I/O Kit port(%08x)\n", kr); return -1; } //Set up matching dictionary for class IOUSBDevice and its subclasses matchingDict = IOServiceMatching(kIOUSBDeviceClassName); if (!matchingDict) { printf("Couldn’t create a USB matching dictionary\n"); mach_port_deallocate(mach_task_self(), masterPort); return -1; } //Add the vendor and product IDs to the matching dictionary. //This is the second key in the table of device-matching keys of the //USB Common Class Specification Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 24CFDictionarySetValue(matchingDict, CFSTR(kUSBVendorName), CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &usbVendor)); CFDictionarySetValue(matchingDict, CFSTR(kUSBProductName), CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &usbProduct)); //To set up asynchronous notifications, create a notification port and //add its run loop event source to the program’s run loop gNotifyPort = IONotificationPortCreate(masterPort); runLoopSource = IONotificationPortGetRunLoopSource(gNotifyPort); CFRunLoopAddSource(CFRunLoopGetCurrent(), runLoopSource, kCFRunLoopDefaultMode); //Retain additional dictionary references because each call to //IOServiceAddMatchingNotification consumes one reference matchingDict = (CFMutableDictionaryRef) CFRetain(matchingDict); matchingDict = (CFMutableDictionaryRef) CFRetain(matchingDict); matchingDict = (CFMutableDictionaryRef) CFRetain(matchingDict); //Now set up two notifications: one to be called when a raw device //is first matched by the I/O Kit and another to be called when the //device is terminated //Notification of first match: kr = IOServiceAddMatchingNotification(gNotifyPort, kIOFirstMatchNotification, matchingDict, RawDeviceAdded, NULL, &gRawAddedIter); //Iterate over set of matching devices to access already-present devices //and to arm the notification RawDeviceAdded(NULL, gRawAddedIter); //Notification of termination: kr = IOServiceAddMatchingNotification(gNotifyPort, kIOTerminatedNotification, matchingDict, RawDeviceRemoved, NULL, &gRawRemovedIter); Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 25//Iterate over set of matching devices to release each one and to //arm the notification RawDeviceRemoved(NULL, gRawRemovedIter); //Now change the USB product ID in the matching dictionary to match //the one the device will have after the firmware has been downloaded usbProduct = kOurProductIDBulkTest; CFDictionarySetValue(matchingDict, CFSTR(kUSBProductName), CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &usbProduct)); //Now set up two notifications: one to be called when a bulk test device //is first matched by the I/O Kit and another to be called when the //device is terminated. //Notification of first match kr = IOServiceAddMatchingNotification(gNotifyPort, kIOFirstMatchNotification, matchingDict, BulkTestDeviceAdded, NULL, &gBulkTestAddedIter); //Iterate over set of matching devices to access already-present devices //and to arm the notification BulkTestDeviceAdded(NULL, gBulkTestAddedIter); //Notification of termination kr = IOServiceAddMatchingNotification(gNotifyPort, kIOTerminatedNotification, matchingDict, BulkTestDeviceRemoved, NULL, &gBulkTestRemovedIter); //Iterate over set of matching devices to release each one and to //arm the notification. NOTE: this function is not shown in this document. BulkTestDeviceRemoved(NULL, gBulkTestRemovedIter); //Finished with master port mach_port_deallocate(mach_task_self(), masterPort); masterPort = 0; Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 26//Start the run loop so notifications will be received CFRunLoopRun(); //Because the run loop will run forever until interrupted, //the program should never reach this point return 0; } Working With the Raw Device Now that you’ve obtained an iterator for a set of matching devices, you can use it to gain access to each raw device, configure it, and download the appropriate firmware to it. The function RawDeviceAdded (shown in Listing 2-3 (page 27)) uses I/O Kit functions to create a device interface for each device and then calls the following functions to configure the device and download firmware to it. ● ConfigureDevice, shown in Listing 2-5 (page 30), uses device interface functions to get the number of configurations, examine the first one, and set the device’s configuration. ● DownloadToDevice, shown in Listing 2-6 (page 32), downloads the firmware in bulktest.c to the device. Listing 2-3 Accessing and programming the raw device void RawDeviceAdded(void *refCon, io_iterator_t iterator) { kern_return_t kr; io_service_t usbDevice; IOCFPlugInInterface **plugInInterface = NULL; IOUSBDeviceInterface **dev = NULL; HRESULT result; SInt32 score; UInt16 vendor; UInt16 product; UInt16 release; while (usbDevice = IOIteratorNext(iterator)) { //Create an intermediate plug-in Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 27kr = IOCreatePlugInInterfaceForService(usbDevice, kIOUSBDeviceUserClientTypeID, kIOCFPlugInInterfaceID, &plugInInterface, &score); //Don’t need the device object after intermediate plug-in is created kr = IOObjectRelease(usbDevice); if ((kIOReturnSuccess != kr) || !plugInInterface) { printf("Unable to create a plug-in (%08x)\n", kr); continue; } //Now create the device interface result = (*plugInInterface)->QueryInterface(plugInInterface, CFUUIDGetUUIDBytes(kIOUSBDeviceInterfaceID), (LPVOID *)&dev); //Don’t need the intermediate plug-in after device interface //is created (*plugInInterface)->Release(plugInInterface); if (result || !dev) { printf("Couldn’t create a device interface (%08x)\n", (int) result); continue; } //Check these values for confirmation kr = (*dev)->GetDeviceVendor(dev, &vendor); kr = (*dev)->GetDeviceProduct(dev, &product); kr = (*dev)->GetDeviceReleaseNumber(dev, &release); if ((vendor != kOurVendorID) || (product != kOurProductID) || (release != 1)) { printf("Found unwanted device (vendor = %d, product = %d)\n", vendor, product); Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 28(void) (*dev)->Release(dev); continue; } //Open the device to change its state kr = (*dev)->USBDeviceOpen(dev); if (kr != kIOReturnSuccess) { printf("Unable to open device: %08x\n", kr); (void) (*dev)->Release(dev); continue; } //Configure device kr = ConfigureDevice(dev); if (kr != kIOReturnSuccess) { printf("Unable to configure device: %08x\n", kr); (void) (*dev)->USBDeviceClose(dev); (void) (*dev)->Release(dev); continue; } //Download firmware to device kr = DownloadToDevice(dev); if (kr != kIOReturnSuccess) { printf("Unable to download firmware to device: %08x\n", kr); (void) (*dev)->USBDeviceClose(dev); (void) (*dev)->Release(dev); continue; } //Close this device and release object kr = (*dev)->USBDeviceClose(dev); kr = (*dev)->Release(dev); Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 29} } The function RawDeviceRemoved simply uses the iterator obtained from the main function (shown in Listing 2-2 (page 24)) to release each device object. This also has the effect of arming the raw device termination notification so it will notify the program of future device removals. RawDeviceRemoved is shown in Listing 2-4 (page 30). Listing 2-4 Releasing the raw device objects void RawDeviceRemoved(void *refCon, io_iterator_t iterator) { kern_return_t kr; io_service_t object; while (object = IOIteratorNext(iterator)) { kr = IOObjectRelease(object); if (kr != kIOReturnSuccess) { printf("Couldn’t release raw device object: %08x\n", kr); continue; } } } Although every USB device has one or more configurations, unless the device is a composite class device that’s been matched by the AppleUSBComposite driver which automatically sets the first configuration, none of those configurations may have been set. Therefore, your application may have to use device interface functions to get the appropriate configuration value and use it to set the device’s configuration. In the sample code, the function ConfigureDevice (shown in Listing 2-5 (page 30)) accomplishes this task. In fact, it is called twice: once by RawDeviceAdded to configure the raw device and again by BulkTestDeviceAdded (shown in Listing 2-7 (page 34)) to configure the bulk test device. Listing 2-5 Configuring a USB device IOReturn ConfigureDevice(IOUSBDeviceInterface **dev) Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 30{ UInt8 numConfig; IOReturn kr; IOUSBConfigurationDescriptorPtr configDesc; //Get the number of configurations. The sample code always chooses //the first configuration (at index 0) but your code may need a //different one kr = (*dev)->GetNumberOfConfigurations(dev, &numConfig); if (!numConfig) return -1; //Get the configuration descriptor for index 0 kr = (*dev)->GetConfigurationDescriptorPtr(dev, 0, &configDesc); if (kr) { printf("Couldn’t get configuration descriptor for index %d (err = %08x)\n", 0, kr); return -1; } //Set the device’s configuration. The configuration value is found in //the bConfigurationValue field of the configuration descriptor kr = (*dev)->SetConfiguration(dev, configDesc->bConfigurationValue); if (kr) { printf("Couldn’t set configuration to value %d (err = %08x)\n", 0, kr); return -1; } return kIOReturnSuccess; } Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 31Now that the device is configured, you can download firmware to it. Cypress makes firmware available to program the EZ-USB chip to emulate different devices. The sample code in this document uses firmware that programs the chip to be a bulk test device, a device that takes the data it receives from its bulk out pipe and echoesit to its bulk in pipe. The firmware, contained in the file bulktest.c, is an array of INTEL_HEX_RECORD structures (defined in the file hex2c.h). The function DownloadToDevice uses the function WriteToDevice (shown together in Listing 2-6 (page 32)) to prepare the device to receive the download and then to write information from each structure to the appropriate address on the device. When all the firmware has been downloaded, DownloadToDevice calls WriteToDevice a last time to inform the device that the download is complete. At this point, the raw device detaches itself from the bus and reattaches as a bulk test device. This causes the device nub representing the raw device to be removed from the I/O Registry and a new device nub, representing the bulk test device, to be attached. Listing 2-6 Two functions to download firmware to the raw device IOReturn DownloadToDevice(IOUSBDeviceInterface **dev) { int i; UInt8 writeVal; IOReturn kr; //Assert reset. This tells the device that the download is //about to occur writeVal = 1; //For this device, a value of 1 indicates a download kr = WriteToDevice(dev, k8051_USBCS, 1, &writeVal); if (kr != kIOReturnSuccess) { printf("WriteToDevice reset returned err 0x%x\n", kr); (*dev)->USBDeviceClose(dev); (*dev)->Release(dev); return kr; } //Download firmware i = 0; while (bulktest[i].Type == 0) //While bulktest[i].Type == 0, this is Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 32{ //not the last firmware record to //download kr = WriteToDevice(dev, bulktest[i].Address, bulktest[i].Length, bulktest[i].Data); if (kr != kIOReturnSuccess) { printf("WriteToDevice download %i returned err 0x%x\n", i, kr); (*dev)->USBDeviceClose(dev); (*dev)->Release(dev); return kr; } i++; } //De-assert reset. This tells the device that the download is complete writeVal = 0; kr = WriteToDevice(dev, k8051_USBCS, 1, &writeVal); if (kr != kIOReturnSuccess) printf("WriteToDevice run returned err 0x%x\n", kr); return kr; } IOReturn WriteToDevice(IOUSBDeviceInterface **dev, UInt16 deviceAddress, UInt16 length, UInt8 writeBuffer[]) { IOUSBDevRequest request; request.bmRequestType = USBmakebmRequestType(kUSBOut, kUSBVendor, kUSBDevice); request.bRequest = 0xa0; request.wValue = deviceAddress; request.wIndex = 0; Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 33request.wLength = length; request.pData = writeBuffer; return (*dev)->DeviceRequest(dev, &request); } Working With the Bulk Test Device After you download the firmware to the device, the raw device is no longer attached to the bus. To gain access to the bulk test device, you repeat most of the same steps you used to get access to the raw device. ● Use the iterator obtained by a call to IOServiceAddMatchingNotification in the main function (shown in Listing 2-2 (page 24)) to iterate over a set of matching devices. ● Create a device interface for each device. ● Configure the device. This time, however, the next step is to find the interfaces on the device so you can choose the appropriate one and get access to its pipes. Because of the similarities of these tasks, the function BulkTestDeviceAdded follows the same outline of the RawDeviceAdded function except that instead of downloading firmware to the device, it calls FindInterfaces (shown in Listing 2-8 (page 36)) to examine the available interfaces and their pipes. The code in Listing 2-7 (page 34) replaces most of the BulkTestDeviceAdded function’s code with comments, focusing on the differences between it and the RawDeviceAdded function. Listing 2-7 Accessing the bulk test device void BulkTestDeviceAdded(void *refCon, io_iterator_t iterator) { kern_return_t kr; io_service_t usbDevice; IOUSBDeviceInterface **device=NULL; while (usbDevice = IOIteratorNext(iterator)) { //Create an intermediate plug-in using the //IOCreatePlugInInterfaceForService function //Release the device object after getting the intermediate plug-in Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 34//Create the device interface using the QueryInterface function //Release the intermediate plug-in object //Check the vendor, product, and release number values to //confirm we’ve got the right device //Open the device before configuring it kr = (*device)->USBDeviceOpen(device); //Configure the device by calling ConfigureDevice //Close the device and release the device interface object if //the configuration is unsuccessful //Get the interfaces kr = FindInterfaces(device); if (kr != kIOReturnSuccess) { printf("Unable to find interfaces on device: %08x\n", kr); (*device)->USBDeviceClose(device); (*device)->Release(device); continue; } //If using synchronous IO, close and release the device interface here #ifndef USB_ASYNC_IO kr = (*device)->USBDeviceClose(device); kr = (*device)->Release(device); #endif } } Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 35The function BulkTestDeviceRemoved simply uses the iterator obtained from the main function (shown in Listing 2-2 (page 24)) to release each device object. This also has the effect of arming the bulk test device termination notification so it will notify the program of future device removals.The BulkTestDeviceRemoved function is identical to the RawDeviceRemoved function (shown in Listing 2-4 (page 30)), with the exception of the wording of the printed error statement. Working With Interfaces Now that you’ve configured the device, you have access to its interfaces. The FindInterfaces function (shown in Listing 2-8 (page 36)) creates an iterator to iterate over all interfaces on the device and then creates a device interface to communicate with each one. For each interface found, the function opens the interface, determines how many endpoints (or pipes) it has, and prints out the properties of each pipe. Because opening an interface causes its pipes to be instantiated, you can get access to any pipe by using its pipe index. The pipe index is the number of the pipe within the interface, ranging from one to the number of endpoints returned by GetNumEndpoints. You can communicate with the default control pipe (described in “USB Transfer Types” (page 8)) from any interface by using pipe index 0, but it is usually better to use the device interface functions for the device itself (see the use of IOUSBDeviceInterface functions in Listing 2-5 (page 30)). The sample code employs conditional compilation using #ifdef and #ifndef to demonstrate both synchronous and asynchronous I/O. If you’ve chosen to test synchronous I/O, FindInterfaces writes the test message (defined in Listing 2-1 (page 22)) to pipe index 2 on the device and readsits echo before returning. For asynchronous I/O, FindInterfaces first creates an event source and adds it to the run loop created by the main function (shown in Listing 2-2 (page 24)). It then sets up an asynchronous write and read that will cause a notification to be sent upon completion. The completion functions WriteCompletion and ReadCompletion are shown together in Listing 2-9 (page 43). Listing 2-8 Finding interfaces on the bulk test device IOReturn FindInterfaces(IOUSBDeviceInterface **device) { IOReturn kr; IOUSBFindInterfaceRequest request; io_iterator_t iterator; io_service_t usbInterface; IOCFPlugInInterface **plugInInterface = NULL; IOUSBInterfaceInterface **interface = NULL; HRESULT result; SInt32 score; Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 36UInt8 interfaceClass; UInt8 interfaceSubClass; UInt8 interfaceNumEndpoints; int pipeRef; #ifndef USE_ASYNC_IO UInt32 numBytesRead; UInt32 i; #else CFRunLoopSourceRef runLoopSource; #endif //Placing the constant kIOUSBFindInterfaceDontCare into the following //fields of the IOUSBFindInterfaceRequest structure will allow you //to find all the interfaces request.bInterfaceClass = kIOUSBFindInterfaceDontCare; request.bInterfaceSubClass = kIOUSBFindInterfaceDontCare; request.bInterfaceProtocol = kIOUSBFindInterfaceDontCare; request.bAlternateSetting = kIOUSBFindInterfaceDontCare; //Get an iterator for the interfaces on the device kr = (*device)->CreateInterfaceIterator(device, &request, &iterator); while (usbInterface = IOIteratorNext(iterator)) { //Create an intermediate plug-in kr = IOCreatePlugInInterfaceForService(usbInterface, kIOUSBInterfaceUserClientTypeID, kIOCFPlugInInterfaceID, &plugInInterface, &score); //Release the usbInterface object after getting the plug-in kr = IOObjectRelease(usbInterface); if ((kr != kIOReturnSuccess) || !plugInInterface) { printf("Unable to create a plug-in (%08x)\n", kr); Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 37break; } //Now create the device interface for the interface result = (*plugInInterface)->QueryInterface(plugInInterface, CFUUIDGetUUIDBytes(kIOUSBInterfaceInterfaceID), (LPVOID *) &interface); //No longer need the intermediate plug-in (*plugInInterface)->Release(plugInInterface); if (result || !interface) { printf("Couldn’t create a device interface for the interface (%08x)\n", (int) result); break; } //Get interface class and subclass kr = (*interface)->GetInterfaceClass(interface, &interfaceClass); kr = (*interface)->GetInterfaceSubClass(interface, &interfaceSubClass); printf("Interface class %d, subclass %d\n", interfaceClass, interfaceSubClass); //Now open the interface. This will cause the pipes associated with //the endpoints in the interface descriptor to be instantiated kr = (*interface)->USBInterfaceOpen(interface); if (kr != kIOReturnSuccess) { printf("Unable to open interface (%08x)\n", kr); (void) (*interface)->Release(interface); break; Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 38} //Get the number of endpoints associated with this interface kr = (*interface)->GetNumEndpoints(interface, &interfaceNumEndpoints); if (kr != kIOReturnSuccess) { printf("Unable to get number of endpoints (%08x)\n", kr); (void) (*interface)->USBInterfaceClose(interface); (void) (*interface)->Release(interface); break; } printf("Interface has %d endpoints\n", interfaceNumEndpoints); //Access each pipe in turn, starting with the pipe at index 1 //The pipe at index 0 is the default control pipe and should be //accessed using (*usbDevice)->DeviceRequest() instead for (pipeRef = 1; pipeRef <= interfaceNumEndpoints; pipeRef++) { IOReturn kr2; UInt8 direction; UInt8 number; UInt8 transferType; UInt16 maxPacketSize; UInt8 interval; char *message; kr2 = (*interface)->GetPipeProperties(interface, pipeRef, &direction, &number, &transferType, &maxPacketSize, &interval); if (kr2 != kIOReturnSuccess) printf("Unable to get properties of pipe %d (%08x)\n", pipeRef, kr2); else Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 39{ printf("PipeRef %d: ", pipeRef); switch (direction) { case kUSBOut: message = "out"; break; case kUSBIn: message = "in"; break; case kUSBNone: message = "none"; break; case kUSBAnyDirn: message = "any"; break; default: message = "???"; } printf("direction %s, ", message); switch (transferType) { case kUSBControl: message = "control"; break; case kUSBIsoc: message = "isoc"; break; case kUSBBulk: message = "bulk"; break; case kUSBInterrupt: message = "interrupt"; Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 40break; case kUSBAnyType: message = "any"; break; default: message = "???"; } printf("transfer type %s, maxPacketSize %d\n", message, maxPacketSize); } } #ifndef USE_ASYNC_IO //Demonstrate synchronous I/O kr = (*interface)->WritePipe(interface, 2, kTestMessage, strlen(kTestMessage)); if (kr != kIOReturnSuccess) { printf("Unable to perform bulk write (%08x)\n", kr); (void) (*interface)->USBInterfaceClose(interface); (void) (*interface)->Release(interface); break; } printf("Wrote \"%s\" (%ld bytes) to bulk endpoint\n", kTestMessage, (UInt32) strlen(kTestMessage)); numBytesRead = sizeof(gBuffer) - 1; //leave one byte at the end //for NULL termination kr = (*interface)->ReadPipe(interface, 9, gBuffer, &numBytesRead); if (kr != kIOReturnSuccess) { printf("Unable to perform bulk read (%08x)\n", kr); (void) (*interface)->USBInterfaceClose(interface); (void) (*interface)->Release(interface); Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 41break; } //Because the downloaded firmware echoes the one’s complement of the //message, now complement the buffer contents to get the original data for (i = 0; i < numBytesRead; i++) gBuffer[i] = ~gBuffer[i]; printf("Read \"%s\" (%ld bytes) from bulk endpoint\n", gBuffer, numBytesRead); #else //Demonstrate asynchronous I/O //As with service matching notifications, to receive asynchronous //I/O completion notifications, you must create an event source and //add it to the run loop kr = (*interface)->CreateInterfaceAsyncEventSource( interface, &runLoopSource); if (kr != kIOReturnSuccess) { printf("Unable to create asynchronous event source (%08x)\n", kr); (void) (*interface)->USBInterfaceClose(interface); (void) (*interface)->Release(interface); break; } CFRunLoopAddSource(CFRunLoopGetCurrent(), runLoopSource, kCFRunLoopDefaultMode); printf("Asynchronous event source added to run loop\n"); bzero(gBuffer, sizeof(gBuffer)); strcpy(gBuffer, kTestMessage); kr = (*interface)->WritePipeAsync(interface, 2, gBuffer, strlen(gBuffer), WriteCompletion, (void *) interface); if (kr != kIOReturnSuccess) Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 42{ printf("Unable to perform asynchronous bulk write (%08x)\n", kr); (void) (*interface)->USBInterfaceClose(interface); (void) (*interface)->Release(interface); break; } #endif //For this test, just use first interface, so exit loop break; } return kr; } When an asynchronous write action is complete, the WriteCompletion function is called by the notification. WriteCompletion then calls the interface function ReadPipeAsync to perform an asynchronous read from the pipe. When the read is complete, control passes to ReadCompletion which simply prints status messages and adds a NULL termination to the global buffer containing the test message read from the device. The WriteCompletion and ReadCompletion functions are shown together in Listing 2-9 (page 43). Listing 2-9 Two asynchronous I/O completion functions void WriteCompletion(void *refCon, IOReturn result, void *arg0) { IOUSBInterfaceInterface **interface = (IOUSBInterfaceInterface **) refCon; UInt32 numBytesWritten = (UInt32) arg0; UInt32 numBytesRead; printf("Asynchronous write complete\n"); if (result != kIOReturnSuccess) { printf("error from asynchronous bulk write (%08x)\n", result); (void) (*interface)->USBInterfaceClose(interface); (void) (*interface)->Release(interface); return; } Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 43printf("Wrote \"%s\" (%ld bytes) to bulk endpoint\n", kTestMessage, numBytesWritten); numBytesRead = sizeof(gBuffer) - 1; //leave one byte at the end for //NULL termination result = (*interface)->ReadPipeAsync(interface, 9, gBuffer, numBytesRead, ReadCompletion, refCon); if (result != kIOReturnSuccess) { printf("Unable to perform asynchronous bulk read (%08x)\n", result); (void) (*interface)->USBInterfaceClose(interface); (void) (*interface)->Release(interface); return; } } void ReadCompletion(void *refCon, IOReturn result, void *arg0) { IOUSBInterfaceInterface **interface = (IOUSBInterfaceInterface **) refCon; UInt32 numBytesRead = (UInt32) arg0; UInt32 i; printf("Asynchronous bulk read complete\n"); if (result != kIOReturnSuccess) { printf("error from async bulk read (%08x)\n", result); (void) (*interface)->USBInterfaceClose(interface); (void) (*interface)->Release(interface); return; } //Check the complement of the buffer’s contents for original data for (i = 0; i < numBytesRead; i++) gBuffer[i] = ~gBuffer[i]; Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 44printf("Read \"%s\" (%ld bytes) from bulk endpoint\n", gBuffer, numBytesRead); } Working With USB Device Interfaces Accessing a USB Device 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 45This table describes the changes to USB Device Interface Guide . Date Notes 2012-01-09 Added information about App Sandbox. 2007-09-04 Made minor corrections. Described how to determine which version of an interface object to use when accessing a USB device or interface. 2007-02-08 2006-04-04 Made minor corrections. Emphasized which type of device interface to get for USB devices and interfaces and clarified definition of composite class device. 2006-03-08 2005-11-09 Made minor corrections. Added information about creating a universal binary for an application that accesses a USB device. 2005-09-08 2005-08-11 Made minor bug fixes. Added information about low latency isochronous transactions and functions. 2005-06-04 Included discussion of USB 2.0 and associated changes to isochronous functions. Changed title from "Working With USB Device Interfaces." 2005-04-29 2004-05-27 Fixed URL for USB Common Class Specification. 2002-11-15 First version. 2012-01-09 | © 2002, 2012 Apple Inc. All Rights Reserved. 46 Document Revision HistoryApple Inc. © 2002, 2012 Apple Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrievalsystem, or transmitted, in any form or by any means, mechanical, electronic, photocopying, recording, or otherwise, without prior written permission of Apple Inc., with the following exceptions: Any person is hereby authorized to store documentation on a single computer for personal use only and to print copies of documentation for personal use provided that the documentation contains Apple’s copyright notice. No licenses, express or implied, are granted with respect to any of the technology described in this document. Apple retains all intellectual property rights associated with the technology described in this document. This document is intended to assist application developers to develop applications only for Apple-labeled computers. Apple Inc. 1 Infinite Loop Cupertino, CA 95014 408-996-1010 Apple, the Apple logo, Finder, Mac, Macintosh, OS X, Pages, Sand, and Xcode are trademarks of Apple Inc., registered in the U.S. and other countries. Intel and Intel Core are registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. PowerPC and the PowerPC logo are trademarks of International Business Machines Corporation, used under license therefrom. Even though Apple has reviewed this document, APPLE MAKES NO WARRANTY OR REPRESENTATION, EITHER EXPRESS OR IMPLIED, WITH RESPECT TO THIS DOCUMENT, ITS QUALITY, ACCURACY, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.ASARESULT, THISDOCUMENT IS PROVIDED “AS IS,” AND YOU, THE READER, ARE ASSUMING THE ENTIRE RISK AS TO ITS QUALITY AND ACCURACY. IN NO EVENT WILL APPLE BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL,OR CONSEQUENTIAL DAMAGES RESULTING FROM ANY DEFECT OR INACCURACY IN THIS DOCUMENT, even if advised of the possibility of such damages. THE WARRANTY AND REMEDIES SET FORTH ABOVE ARE EXCLUSIVE AND IN LIEU OF ALL OTHERS, ORAL OR WRITTEN, EXPRESS OR IMPLIED. No Apple dealer, agent, or employee is authorized to make any modification, extension, or addition to this warranty. Some states do not allow the exclusion or limitation of implied warranties or liability for incidental or consequential damages, so the above limitation or exclusion may not apply to you. This warranty gives you specific legal rights, and you may also have other rights which vary from state to state. Core Data Model Versioning and Data Migration Programming GuideContents Core Data Model Versioning and Data Migration 5 At a Glance 5 Prerequisites 6 Understanding Versions 7 Model File Format and Versions 10 Lightweight Migration 12 Core Data Must Be Able to Infer the Mapping 12 Request Automatic Migration Using an Options Dictionary 13 Use a Migration Manager if Models Cannot Be Found Automatically 14 Mapping Overview 17 Mapping Model Objects 17 Creating a Mapping Model in Xcode 19 The Migration Process 20 Overview 20 Requirements for the Migration Process 20 Custom Entity Migration Policies 21 Three-Stage Migration 21 Initiating the Migration Process 23 Initiating the Migration Process 23 The Default Migration Process 24 Customizing the Migration Process 26 Is Migration Necessary 26 Initializing a Migration Manager 27 Performing a Migration 28 Multiple Passes—Dealing With Large Datasets 29 Migration and iCloud 30 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 2Document Revision History 31 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 3 ContentsFigures and Listings Understanding Versions 7 Figure 1-1 Recipes models “Version 1.0” 7 Figure 1-2 Recipes model “Version 1.1” 7 Figure 1-3 Recipes model “Version 2.0” 8 Model File Format and Versions 10 Figure 2-1 Initial version of the Core Recipes model 10 Figure 2-2 Version 2 of the Core Recipes model 11 Mapping Overview 17 Figure 4-1 Mapping model for versions 1-2 of the Core Recipes models 19 Initiating the Migration Process 23 Listing 6-1 Opening a store using automatic migration 24 Customizing the Migration Process 26 Listing 7-1 Checking whether migration is necessary 26 Listing 7-2 Initializing a Migration Manager 27 Listing 7-3 Performing a Migration 28 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 4Core Data provides support for managing changes to a managed object model as your application evolves. You can only open a Core Data store using the managed object model used to create it. Changing a model will therefore make it incompatible with (and so unable to open) the stores it previously created. If you change your model, you therefore need to change the data in existing stores to new version—changing the store format is known as migration. To migrate a store, you need both the version of the model used to create it, and the current version of the model you want to migrate to. You can create a versioned model that contains more than one version of a managed object model. Within the versioned model you mark one version as being the current version. Core Data can then use this model to open persistent stores created using any of the model versions, and migrate the stores to the current version. To help Core Data perform the migration, though, you may have to provide information about how to map from one version of the model to another. This information may be in the form of hints within the versioned model itself, or in a separate mapping model file that you create. At a Glance Typically, as it evolves from one version to another, numerous aspects of your application change: the classes you implement, the user interface, the file format, and so on. You need to be aware of and in control of all these aspects; there is no API that solves the problems associated with all these—for example Cocoa does not provide a means to automatically update your user interface if you add a new attribute to an entity in your managed object model. Core Data does not solve all the issues of how you roll out your application. It does, though, provide support for a small—but important and non-trivial—subset of the tasks you must perform as your application evolves. ● Model versioning allows you to specify and distinguish between different configurations of your schema. There are two distinct views of versioning: your perspective as a developer, and Core Data’s perspective. These may not always be the same. The differences are discussed in “Understanding Versions” (page 7). The format of a versioned managed object model, and how you add a version to a model, is discussed in “Model File Format and Versions” (page 10). ● Core Data needs to know how to map from the entities and properties in a source model to the entities and properties in the destination model. 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 5 Core Data Model Versioning and Data MigrationIn many cases, Core Data can infer the mapping from existing versions of the managed object model. This is described in “Lightweight Migration” (page 12). If you make changes to your models such that Core Data cannot infer the mapping from source to destination, you need to create a mapping model. A mapping model parallels a managed object model, specifying how to transform objects in the source into instances appropriate for the destination. How you create a mapping model is discussed in “Mapping Overview” (page 17). ● Data migration allows you to convert data from one model (schema) to another, using mappings. The migration process itself is discussed in “The Migration Process” (page 20). How you perform a migration is discussed in “Initiating the Migration Process” (page 23). You can also customize the migration process—that is, how you programmatically determine whether migration is necessary; how you find the correct source and destination models and the appropriate mapping model to initialize the migration manager; and then how you perform the migration. You only customize the migration process if you want to initiate migration yourself. You might do this to, for example, search locations other than the application’s main bundle for models or to deal with large data sets by performing the migration in several passes using different mapping models. How you can customize the process is described in “Customizing the Migration Process” (page 26). ● If you are using iCloud, there are some constraints on what migration you can perform. If you are using iCloud, you must use lightweight migration. Other factors to be aware of are described in “Migration and iCloud” (page 30). Although Core Data makes versioning and migration easier than would typically otherwise be the case, these processes are still non-trivial in effect. You still need to carefully consider the implications of releasing and supporting different versions of your application. Prerequisites This document assumes that you are familiar with the Core Data architecture and the fundamentals of using Core Data. You should be able to identify the parts of the Core Data stack and understand the roles of the model, the managed object context, and the persistent store coordinator. You need to know how to create a managed object model, how to create and programmatically interact with parts of the Core Data stack. If you do not meet these requirements, you should first read the Core Data Programming Guide and related materials. You are strongly encouraged also to work through the Core Data Utility Tutorial . Core Data Model Versioning and Data Migration Prerequisites 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 6There are two distinct views of versioning: your perspective as a developer, and Core Data’s perspective. These may not always be the same—consider the following models. Figure 1-1 Recipes models “Version 1.0” Recipe Attributes cuisine directions name Relationships chef ingredients Chef Attributes name training Relationships recipes Ingredient Attributes amount name Relationships recipes Figure 1-2 Recipes model “Version 1.1” Recipe Attributes cuisine directions name Relationships chef ingredients Chef Attributes name training Relationships recipes Ingredient Attributes amount name Relationships recipes Recipe changes: • Add validation rules • Change User Info values • Use custom class 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 7 Understanding VersionsFigure 1-3 Recipes model “Version 2.0” Recipe Attributes directions name rating Relationships chef cuisines ingredients Chef Attributes firstName lastName Relationships recipes Ingredient Attributes amount name Relationships recipe Cuisine Attributes name Relationships recipes As a developer, your perspective is typically that a version is denoted by an identifier—a string or number, such as “9A218”, “2.0.7”, or “Version 1.1”. To support this view, managed object models have a set of identifiers (see versionIdentifiers)—typically for a single model you provide a single string (the attribute itself is a set so that if models are merged all the identifiers can be preserved). How the identifier should be interpreted is up to you, whether it represents the version number of the application, the version that was committed prior to going on vacation, or the last submission before it stopped working. Core Data, on the other hand, treats these identifiers simply as “hints”. To understand why, recall that the format of a persistent store is dependent upon the model used to create it, and that to open a persistent store you must have a model that is compatible with that used to create it. Consider then what would happen if you changed the model but not the identifier—for example, if you kept the identifier the same but removed one entity and added two others. To Core Data, the change in the schema is significant, the fact that the identifier did not change is irrelevant. Core Data’s perspective on versioning isthat it is only interested in features of the model that affect persistence. This means that for two models to be compatible: ● For each entity the following attributes must be equal: name, parent, isAbstract, and properties. className, userInfo, and validation predicates are not compared. ● For each property in each entity, the following attributes must be equal: name, isOptional, isTransient, isReadOnly, for attributes attributeType, and for relationships destinationEntity, minCount, maxCount, deleteRule, and inverseRelationship. userInfo and validation predicates are not compared. Notice that Core Data ignores any identifiers you set. In the examples above, Core Data treats version 1.0 (Figure 1-1 (page 7)) and 1.1 (Figure 1-2 (page 7)) as being compatible. Understanding Versions 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 8Rather than enumerating through all the relevant parts of a model, Core Data creates a 32-byte hash digest of the components which it compares for equality (see versionHash (NSEntityDescription) and versionHash (NSPropertyDescription)). These hashes are included in a store’s metadata so that Core Data can quickly determine whether the store format matches that of the managed object model it may use to try to open the store. (When you attempt to open a store using a given model, Core Data compares the version hashes of each of the entities in the store with those of the entities in the model, and if all are the same then the store is opened.) There is typically no reason for you to be interested in the value of a hash. There may, however, be some situations in which you have two versions of a model that Core Data would normally treat as equivalent that you want to be recognized as being different. For example, you might change the name of the class used to represent an entity, or more subtly you might keep the model the same but change the internal format of an attribute such as a BLOB—this is irrelevant to Core Data, but it is crucial for the integrity of your data. To support this, Core Data allows you to set a hash modifier for an entity or property see versionHashModifier (NSEntityDescription) and versionHashModifier (NSPropertyDescription). In the examples above, if you wanted to force Core Data to recognize that “Version 1.0” (Figure 1-1 (page 7)) and “Version 1.1” (Figure 1-2 (page 7)) of your models are different, you could set an entity modifier for the Recipe entity in the second model to change the version hash Core Data creates. Understanding Versions 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 9A managed object model that supports versioning is represented in the filesystem by a .xcdatamodeld document. An .xcdatamodeld document is a file package (see “Document Packages”) that groups versions of the model, each represented by an individual .xcdatamodel file, and an Info.plist file that contains the version information. The model is compiled into a runtime format—a file package with a .momd extension that containsindividually compiled model files with a .mom extension. You load the .momd model bundle using NSManagedObjectModel’s initWithContentsOfURL:. To add a version to a model, you start with a model such as that illustrated in Figure 2-1. Figure 2-1 Initial version of the Core Recipes model 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 10 Model File Format and VersionsTo add a version, select Editor > Add Model Version. In the sheet that appears, you enter the name of the new model version and select the model on which it should be based. To set the new model asthe current version of the model,select the .xcdatamodeld document in the project navigator, then select the new model in the pop-up menu in the Versioned Core Data Model area in the Attributes Inspector (see Figure 2-2). Figure 2-2 Version 2 of the Core Recipes model Model File Format and Versions 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 11If you just make simple changes to your model (such as adding a new attribute to an entity), Core Data can perform automatic data migration, referred to aslightweightmigration. Lightweight migration isfundamentally the same as ordinary migration, except that instead of you providing a mapping model (as described in “Mapping Overview” (page 17)), Core Data infers one from differences between the source and destination managed object models. Lightweight migration is especially convenient during early stages of application development, when you may be changing your managed object model frequently, but you don’t want to have to keep regenerating test data. You can migrate existing data without having to create a custom mapping model for every model version used to create a store that would need to be migrated. A further advantage of using lightweight migration—beyond the fact that you don’t need to create the mapping model yourself—is that if you use an inferred model and you use the SQLite store, then Core Data can perform the migration in situ (solely by issuing SQL statements). This can represent a significant performance benefit as Core Data doesn’t have to load any of your data. Because of this, you are encouraged to use inferred migration where possible, even if the mapping model you might create yourself would be trivial. Core Data Must Be Able to Infer the Mapping To perform automatic lightweight migration, Core Data needs to be able to find the source and destination managed object models itself at runtime. Core Data looks for models in the bundles returned by NSBundle’s allBundles and allFrameworks methods. If you store your models elsewhere, you must follow the steps described in “Use a Migration Manager if Models Cannot Be Found Automatically ” (page 14). Core Data must then analyze the schema changes to persistent entities and properties and generate an inferred mapping model. For Core Data to be able to generate an inferred mapping model, changes must fit an obvious migration pattern, for example: ● Simple addition of a new attribute ● Removal of an attribute ● A non-optional attribute becoming optional ● An optional attribute becoming non-optional, and defining a default value 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 12 Lightweight Migration● Renaming an entity or property If you rename an entity or property, you can set the renaming identifier in the destination model to the name of the corresponding property or entity in the source model. You set the renaming identifier in the managed object model using the Xcode Data Modeling tool’s property inspector (for either an entity or a property). For example, you can: ● Rename a Car entity to Automobile ● Rename a Car’s color attribute to paintColor The renaming identifier creates a “canonical name,” so you should set the renaming identifier to the name of the property in the source model (unless that property already has a renaming identifier). This means you can rename a property in version 2 of a model then rename it again version 3, and the renaming will work correctly going from version 2 to version 3 or from version 1 to version 3. In addition, Core Data supports: ● Adding relationships and changing the type of relationship ● You can add a new relationship or delete an existing relationship. ● Renaming a relationship (by using a renaming identifier, just like an attribute) ● Changing a relationship from a to-one to a to-many, or a non-ordered to-many to ordered (and visa-versa) ● Changing the entity hierarchy ● You can add, remove, rename entities ● You can create a new parent or child entity and move properties up and down the entity hierarchy ● You can move entities out of a hierarchy You cannot, however, merge entity hierarchies; if two existing entities do not share a common parent in the source, they cannot share a common parent in the destination Request Automatic Migration Using an Options Dictionary You request automatic lightweight migration using the options dictionary you pass in addPersistentStoreWithType:configuration:URL:options:error:, by setting values corresponding to both the NSMigratePersistentStoresAutomaticallyOption and the NSInferMappingModelAutomaticallyOption keys to YES: NSError *error = nil; Lightweight Migration Request Automatic Migration Using an Options Dictionary 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 13NSURL *storeURL = <#The URL of a persistent store#>; NSPersistentStoreCoordinator *psc = <#The coordinator#>; NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption, [NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil]; BOOL success = [psc addPersistentStoreWithType:<#Store type#> configuration:<#Configuration or nil#> URL:storeURL options:options error:&error]; if (!success) { // Handle the error. } If you want to determine in advance whether Core Data can infer the mapping between the source and destination models without actually doing the work of migration, you can use NSMappingModel’s inferredMappingModelForSourceModel:destinationModel:error: method. Thisreturnsthe inferred model if Core Data is able to create it, otherwise nil. Use a Migration Manager if Models Cannot Be Found Automatically To perform automatic migration, Core Data has to be able to find the source and destination managed object models itself at runtime (see “Core Data Must Be Able to Infer the Mapping” (page 12)). If you need to put your models in the locations not checked by automatic discovery, then you need to generate the inferred model and initiate the migration yourself using a migration manager (an instance of NSMigrationManager). The following code sample illustrates how to generate an inferred model and initiate the migration using a migration manager. The code assumes that you have implemented two methods—sourceModel and destinationModel—that return the source and destination managed object models respectively. - (BOOL)migrateStore:(NSURL *)storeURL toVersionTwoStore:(NSURL *)dstStoreURL error:(NSError **)outError { // Try to get an inferred mapping model. NSMappingModel *mappingModel = [NSMappingModel inferredMappingModelForSourceModel:[self sourceModel] destinationModel:[self destinationModel] error:outError]; Lightweight Migration Use a Migration Manager if Models Cannot Be Found Automatically 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 14// If Core Data cannot create an inferred mapping model, return NO. if (!mappingModel) { return NO; } // Create a migration manager to perform the migration. NSMigrationManager *manager = [[NSMigrationManager alloc] initWithSourceModel:[self sourceModel] destinationModel:[self destinationModel]]; BOOL success = [manager migrateStoreFromURL:storeURL type:NSSQLiteStoreType options:nil withMappingModel:mappingModel toDestinationURL:dstStoreURL destinationType:NSSQLiteStoreType destinationOptions:nil error:outError]; return success; } Lightweight Migration Use a Migration Manager if Models Cannot Be Found Automatically 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 15Note: Prior to OS X v10.7 and iOS 4, you need to use a store-specific migration manager to perform lightweight migration. You get the migration manager for a given persistent store type using migrationManagerClass, as illustrated in the following example. - (BOOL)migrateStore:(NSURL *)storeURL toVersionTwoStore:(NSURL *)dstStoreURL error:(NSError **)outError { // Try to get an inferred mapping model. NSMappingModel *mappingModel = [NSMappingModel inferredMappingModelForSourceModel:[self sourceModel] destinationModel:[self destinationModel] error:outError]; // If Core Data cannot create an inferred mapping model, return NO. if (!mappingModel) { return NO; } // Get the migration manager class to perform the migration. NSValue *classValue = [[NSPersistentStoreCoordinator registeredStoreTypes] objectForKey:NSSQLiteStoreType]; Class sqliteStoreClass = (Class)[classValue pointerValue]; Class sqliteStoreMigrationManagerClass = [sqliteStoreClass migrationManagerClass]; NSMigrationManager *manager = [[sqliteStoreMigrationManagerClass alloc] initWithSourceModel:[self sourceModel] destinationModel:[self destinationModel]]; BOOL success = [manager migrateStoreFromURL:storeURL type:NSSQLiteStoreType options:nil withMappingModel:mappingModel toDestinationURL:dstStoreURL destinationType:NSSQLiteStoreType destinationOptions:nil error:outError]; return success; } Lightweight Migration Use a Migration Manager if Models Cannot Be Found Automatically 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 16In many cases, Core Data may be able to infer how to transform data from one schema to another (see “Lightweight Migration” (page 12). If Core Data cannot infer the mapping from one model to another, you need a definition of how to perform the transformation. This information is captured in a mapping model. A mapping model is a collection of objects that specifies the transformations that are required to migrate part of a store from one version of your model to another (for example, that one entity is renamed, an attribute is added to another, and a third split into two). You typically create a mapping model in Xcode. Much as the managed object model editor allows you to graphically create the model, the mapping model editor allows you to customize the mappings between the source and destination entities and properties. Mapping Model Objects Like a managed object model, a mapping model is a collection of objects. Mapping model classes parallel the managed object model classes—there are mapping classes for a model, an entity, and a property (NSMappingModel, NSEntityMapping, and NSPropertyMapping respectively). ● An instance of NSEntityMapping specifies a source entity, a destination entity (the type of object to create to correspond to the source object) and mapping type (add, remove, copy as is, or transform). ● An instance of NSPropertyMapping specifiesthe name of the property in the source and in the destination entity, and a value expression to create the value for the destination property. The model does not contain instances of NSEntityMigrationPolicy or any of its subclasses, however amongst other attributes instance of NSEntityMapping can specify the name of an entity migration policy class (a subclass of NSEntityMigrationPolicy) to use to customize the migration. For more about entity migration policy classes, see “Custom Entity Migration Policies” (page 21). You can handle simple property migration changes by configuring a custom value expression on a property mapping directly in the mapping model editor in Xcode. For example, you can: ● Migrate data from one attribute to another. To rename amount to totalCost, enter the custom value expression for the totalCost property mapping as $source.amount. ● Apply a value transformation on a property. 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 17 Mapping OverviewTo convert temperature from Fahrenheit to Celsius, use the custom value expression ($source.temperature - 32.0) / 1.8. ● Migrate objects from one relationship to another. To rename trades to transactions, enter the custom value expression for the transactions property mapping as FUNCTION($manager, "destinationInstancesForEntityMappingNamed:sourceInstances:", "TradeToTrade", $source.trades). (This assumes the entity mapping that migrates Trade instances is named TradeToTrade.) There are six predefined keys you can reference in custom value expressions. To access these keys in source code, you use the constants as declared. To access them in custom value expression strings in the mapping model editor in Xcode, follow the syntax rules outlined in the predicate format string syntax guide and refer to them as: NSMigrationManagerKey: $manager NSMigrationSourceObjectKey: $source NSMigrationDestinationObjectKey: $destination NSMigrationEntityMappingKey: $entityMapping NSMigrationPropertyMappingKey: $propertyMapping NSMigrationEntityPolicyKey: $entityPolicy Mapping Overview Mapping Model Objects 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 18Creating a Mapping Model in Xcode From the File menu, you select New File and in the New File pane select Design > Mapping Model. In the following pane, you select the source and destination models. When you click Finish, Xcode creates a new mapping model that contains as many default mappings as it can deduce from the source and destination. For example, given the model files shown in Figure 1-1 (page 7) and Figure 1-2 (page 7), Xcode creates a mapping model as shown in Figure 4-1. Figure 4-1 Mapping model for versions 1-2 of the Core Recipes models Reserved words in custom value expressions: If you use a custom value expression, you must escape reserved words such as SIZE, FIRST, and LAST using a # (for example, $source.#size). Mapping Overview Creating a Mapping Model in Xcode 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 19During migration, Core Data creates two stacks, one for the source store and one for the destination store. Core Data then fetches objects from the source stack and inserts the appropriate corresponding objects into the destination stack. Note that Core Data must re-create objects in the new stack. Overview Recall that stores are bound to their models. Migration is required when the model doesn't match the store. There are two areas where you get default functionality and hooks for customizing the default behavior: ● When detecting version skew and initializing the migration process. ● When performing the migration process. To perform the migration processrequirestwo Core Data stacks—which are automatically created for you—one for the source store, one for the destination store. The migration process is performed in 3 stages, copying objects from one stack to another. Requirements for the Migration Process Migration of a persistent store is performed by an instance of NSMigrationManager. To migrate a store, the migration manager requires several things: ● The managed object model for the destination store. This is the persistent store coordinator’s model. ● A managed object model that it can use to open the existing store. ● Typically, a mapping model that defines a transformation from the source (the store’s) model to the destination model. You don’t need a mapping model if you’re able to use lightweight migration—see “Lightweight Migration” (page 12). 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 20 The Migration ProcessYou can specify custom entity migration policy classes to customize the migration of individual entities. You specify custom migration policy classesin the mapping model (note the “Custom Entity Policy Name” text field in Figure 4-1 (page 19)). Custom Entity Migration Policies If your new model simply adds properties or entities to your existing model, there may be no need to write any custom code. If the transformation is more complex, however, you might need to create a subclass of NSEntityMigrationPolicy to perform the transformation; for example: ● If you have a Person entity that also includes address information that you want to split into a separate Address entity, but you want to ensure uniqueness of each Address. ● If you have an attribute that encodes data in a string format that you want to change to a binary representation. The methods you override in a custom migration policy correspond to the different phases of the migration process—these are called out in the description of the process given in “Three-Stage Migration.” Three-Stage Migration The migration process itself is in three stages. It uses a copy of the source and destination models in which the validation rules are disabled and the class of all entities is changed to NSManagedObject. To perform the migration, Core Data sets up two stacks, one for the source store and one for the destination store. Core Data then processes each entity mapping in the mapping model in turn. It fetches objects of the current entity into the source stack, creates the corresponding objects in the destination stack, then recreates relationships between destination objects in a second stage, before finally applying validation constraints in the final stage. Before a cycle starts, the entity migration policy responsible for the current entity is sent a beginEntityMapping:manager:error: message. You can override this method to perform any initialization the policy requires. The process then proceeds as follows: 1. Create destination instances based on source instances. At the beginning of this phase, the entity migration policy is sent a createDestinationInstancesForSourceInstance:entityMapping:manager:error:message; at the end it is sent a endInstanceCreationForEntityMapping:manager:error: message. In this stage, only attributes (not relationships) are set in the destination objects. The Migration Process Custom Entity Migration Policies 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 21Instances of the source entity are fetched. For each instance, appropriate instances of the destination entity are created (typically there is only one) and their attributes populated (for trivial cases, name = $source.name). A record is kept of the instances per entity mapping since this may be useful in the second stage. 2. Recreate relationships. At the beginning of this phase, the entity migration policy is sent a createRelationshipsForDestinationInstance:entityMapping:manager:error: message; at the end it is sent a endRelationshipCreationForEntityMapping:manager:error: message. For each entity mapping (in order), for each destination instance created in the first step any relationships are recreated. 3. Validate and save. In this phase, the entity migration policy is sent a performCustomValidationForEntityMapping:manager:error: message. Validation rules in the destination model are applied to ensure data integrity and consistency, and then the store is saved. At the end of the cycle, the entity migration policy issent an endEntityMapping:manager:error: message. You can override this method to perform any clean-up the policy needs to do. Note that Core Data cannot simply fetch objects into the source stack and insert them into the destination stack, the objects must be re-created in the new stack. Core Data maintains “association tables” which tell it which object in the destination store isthe migrated version of which object in the source store, and vice-versa. Moreover, because it doesn't have a means to flush the contexts it is working with, you may accumulate many objects in the migration manager as the migration progresses. If this presents a significant memory overhead and hence gives rise to performance problems, you can customize the process as described in “Multiple Passes—Dealing With Large Datasets” (page 29). The Migration Process Three-Stage Migration 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 22This chapter describes how to initiate the migration process and how the default migration process works. It does not describe customizing the migration process—this is described in “Customizing the Migration Process” (page 26). Initiating the Migration Process When you initialize a persistent store coordinator, you assign to it a managed object model (see initWithManagedObjectModel:); the coordinator uses that model to open persistent stores. You open a persistent store using addPersistentStoreWithType:configuration:URL:options:error:. How you use this method, however, depends on whether your application uses model versioning and on how you choose to support migration—whether you choose to use the default migration process or custom version skew detection and migration bootstrapping. The following list describes different scenarios and what you should do in each: ● Your application does not support versioning You use addPersistentStoreWithType:configuration:URL:options:error: directly. If for some reason the coordinator’s model is not compatible with the store schema (that is, the version hashes current model’s entities do not equal those in the store’s metadata), the coordinator detects this, generates an error, and addPersistentStoreWithType:configuration:URL:options:error: returns NO. You must deal with this error appropriately. ● Your application does support versioning and you choose to use either the lightweight or the default migration process You use addPersistentStoreWithType:configuration:URL:options:error: as described in “Lightweight Migration” (page 12) and “The Default Migration Process” (page 24) respectively. The fundamental difference from the non-versioned approach is that you instruct the coordinator to automatically migrate the store to the current model version by adding an entry to the options dictionary where the key is NSMigratePersistentStoresAutomaticallyOption and the value is an NSNumber object that represents YES. ● Your application does support versioning and you choose to use custom version skew detection and migration bootstrapping 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 23 Initiating the Migration ProcessBefore opening a store you use isConfiguration:compatibleWithStoreMetadata: to check whether its schema is compatible with the coordinator’s model: ● If it is, you use addPersistentStoreWithType:configuration:URL:options:error: to open the store directly; ● If it is not, you must migrate the store first then open it (again using addPersistentStoreWithType:configuration:URL:options:error:). You could simply use addPersistentStoreWithType:configuration:URL:options:error: to check whether migration is required, however this is a heavyweight operation and inefficient for this purpose. It is important to realize that there are two orthogonal concepts: 1. You can execute custom code during the migration. 2. You can have custom code for version skew detection and migration bootstrapping. The migration policy classes allow you to customize the migration of entities and properties in a number of ways, and these are typically all you need. You might, however, use custom skew detection and migration bootstrapping so that you can take control of the migration process. For example, if you have very large stores you could set up a migration manager with the two data models, and then use a series of mapping models to migrate your data into your destination store (if you use the same destination URL for each invocation, Core Data adds new objects to the existing store). This allows the framework (and you) to limit the amount of data in memory during the conversion process. The Default Migration Process To open a store and perform migration (if necessary), you use addPersistentStoreWithType:configuration:URL:options:error: and add to the options dictionary an entry where the key is NSMigratePersistentStoresAutomaticallyOption and the value is an NSNumber object that represents YES. Your code looks similar to the following example: Listing 6-1 Opening a store using automatic migration NSError *error; NSPersistentStoreCoordinator *psc = <#The coordinator#>; NSURL *storeURL = <#The URL of a persistent store#>; NSDictionary *optionsDictionary = [NSDictionary dictionaryWithObject:[NSNumber numberWithBool:YES] Initiating the Migration Process The Default Migration Process 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 24forKey:NSMigratePersistentStoresAutomaticallyOption]; NSPersistentStore *store = [psc addPersistentStoreWithType:<#Store type#> configuration:<#Configuration or nil#> URL:storeURL options:optionsDictionary error:&error]; If the migration proceeds successfully, the existing store at storeURL is renamed with a “~” suffix before any file extension and the migrated store saved to storeURL. In its implementation of addPersistentStoreWithType:configuration:URL:options:error: Core Data does the following: 1. Tries to find a managed object model that it can use to open the store. Core Data searches through your application’s resources for models and tests each in turn. If it cannot find a suitable model, Core Data returns nil and a suitable error. 2. Tries to find a mapping model that maps from the managed object model for the existing store to that in use by the persistent store coordinator. Core Data searches through your application’s resources for available mapping models and tests each in turn. If it cannot find a suitable mapping, Core Data returns NO and a suitable error. Note that you must have created a suitable mapping model in order for this phase to succeed. 3. Creates instances of the migration policy objects required by the mapping model. Note that even if you use the default migration process you can customize the migration itself using custom migration policy classes. Initiating the Migration Process The Default Migration Process 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 25You only customize the migration process if you want to initiate migration yourself. You might do this to, for example, to search for models in locations other than the application’s main bundle, or to deal with large data sets by performing the migration in several passes using different mapping models (see “Multiple Passes—Dealing With Large Datasets” (page 29)). Is Migration Necessary Before you initiate a migration process, you should first determine whether it is necessary. You can check with NSManagedObjectModel’s isConfiguration:compatibleWithStoreMetadata: asillustrated in Listing 7-1 (page 26). Listing 7-1 Checking whether migration is necessary NSPersistentStoreCoordinator *psc = /* get a coordinator */ ; NSString *sourceStoreType = /* type for the source store, or nil if not known */ ; NSURL *sourceStoreURL = /* URL for the source store */ ; NSError *error = nil; NSDictionary *sourceMetadata = [NSPersistentStoreCoordinator metadataForPersistentStoreOfType:sourceStoreType URL:sourceStoreURL error:&error]; if (sourceMetadata == nil) { // deal with error } NSString *configuration = /* name of configuration, or nil */ ; NSManagedObjectModel *destinationModel = [psc managedObjectModel]; BOOL pscCompatibile = [destinationModel 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 26 Customizing the Migration ProcessisConfiguration:configuration compatibleWithStoreMetadata:sourceMetadata]; if (pscCompatibile) { // no need to migrate } Initializing a Migration Manager You initialize a migration manager using initWithSourceModel:destinationModel:; you therefore first need to find the appropriate model for the store. You get the model for the store using NSManagedObjectModel’s mergedModelFromBundles:forStoreMetadata:. If this returns a suitable model, you can create the migration manager as illustrated in Listing 7-2 (page 27) (this code fragment continues from Listing 7-1 (page 26)). Listing 7-2 Initializing a Migration Manager NSArray *bundlesForSourceModel = /* an array of bundles, or nil for the main bundle */ ; NSManagedObjectModel *sourceModel = [NSManagedObjectModel mergedModelFromBundles:bundlesForSourceModel forStoreMetadata:sourceMetadata]; if (sourceModel == nil) { // deal with error } MyMigrationManager *migrationManager = [[MyMigrationManager alloc] initWithSourceModel:sourceModel destinationModel:destinationModel]; Customizing the Migration Process Initializing a Migration Manager 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 27Performing a Migration You migrate a store using NSMigrationManager’s migrateStoreFromURL:type:options:withMappingModel:toDestinationURL:destinationType:destinationOptions:error:. To use this method you need to marshal a number of parameters; most are straightforward, the only one that requires some work is the discovery of the appropriate mapping model (which you can retrieve using NSMappingModel’s mappingModelFromBundles:forSourceModel:destinationModel:method). This is illustrated in Listing 7-3 (page 28) (a continuation of the example shown in Listing 7-2 (page 27)). Listing 7-3 Performing a Migration NSArray *bundlesForMappingModel = /* an array of bundles, or nil for the main bundle */ ; NSError *error = nil; NSMappingModel *mappingModel = [NSMappingModel mappingModelFromBundles:bundlesForMappingModel forSourceModel:sourceModel destinationModel:destinationModel]; if (mappingModel == nil) { // deal with the error } NSDictionary *sourceStoreOptions = /* options for the source store */ ; NSURL *destinationStoreURL = /* URL for the destination store */ ; NSString *destinationStoreType = /* type for the destination store */ ; NSDictionary *destinationStoreOptions = /* options for the destination store */ ; BOOL ok = [migrationManager migrateStoreFromURL:sourceStoreURL type:sourceStoreType options:sourceStoreOptions withMappingModel:mappingModel toDestinationURL:destinationStoreURL destinationType:destinationStoreType destinationOptions:destinationStoreOptions Customizing the Migration Process Performing a Migration 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 28error:&error]; Multiple Passes—Dealing With Large Datasets The basic approach shown above is to have the migration manager take two models, and then iterate over the steps (mappings) provided in a mapping model to move the data from one side to the next. Because Core Data performs a "three stage" migration—where it creates all of the data first, and then relates the data in a second stage—it must maintain “association tables" (which tell it which object in the destination store is the migrated version of which object in the source store, and vice-versa). Further, because it doesn't have a means to flush the contexts it is working with, it means you'll accumulate many objects in the migration manager as the migration progresses. In order to address this, the mapping model is given as a parameter of the migrateStoreFromURL:type:options:withMappingModel:toDestinationURL:destinationType:destinationOptions:error: call itself. What this means is that if you can segregate parts of your graph (as far as mappings are concerned) and create them in separate mapping models, you could do the following: 1. Get the source and destination data models 2. Create a migration manager with them 3. Find all of your mapping models, and put them into an array (in some defined order, if necessary) 4. Loop through the array, and call migrateStoreFromURL:type:options:withMappingModel:toDestinationURL:destinationType:destinationOptions:error: with each of the mappings This allows you to migrate "chunks" of data at a time, while not pulling in all of the data at once. From a "tracking/showing progress” point of view, that basically just creates another layer to work from, so you'd be able to determine percentage complete based on number of mapping models to iterate through (and then further on the number of entity mappings in a model you've already gone through). Customizing the Migration Process Multiple Passes—Dealing With Large Datasets 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 29If you are using iCloud, you can only migrate the contents of a store using automatic lightweight migration. To migrate a persistent store that is in iCloud, you add the store to a persistent store coordinator using addPersistentStoreWithType:configuration:URL:options:error: and pass at least the following options in the options dictionary: NSDictionary *optionsDictionary = [[NSDictionary alloc] initWithObjectsAndKeys: [NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, [NSNumber numberWithBool:YES], NSMigratePersistentStoresAutomaticallyOption, <#Ubiquitous content name#>, NSPersistentStoreUbiquitousContentNameKey, nil]; Changes to a store are recorded and preserved independently for each model version that is associated with a given NSPersistentStoreUbiquitousContentNameKey. A persistent store configured with a given NSPersistentStoreUbiquitousContentNameKey only syncs data with a store on another device data if the model versions match. If you migrate a persistent store configured with a NSPersistentStoreUbiquitousContentNameKey option to a new model version, the store’s history of changes originating from the current device will also be migrated and then merged with any other devices configured with that new model version. Any changes from stores using the new version are also merged in. Existing changes can not, however, be migrated to a new model version if the migration is performed using a custom mapping model. 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 30 Migration and iCloudThis table describes the changes to Core Data Model Versioning and Data Migration Programming Guide . Date Notes 2012-01-09 Updated to describe use of migration with iCloud. 2010-02-24 Added further details to the section on Mapping Model Objects. 2009-06-04 Added an article to describe the lightweight migration feature. 2009-03-05 First version for iOS. 2008-02-08 Added a note about migrating stores from OS X v10.4 (Tiger). New document that describes managed object model versioning and Core Data migration. 2007-05-18 2012-01-09 | © 2012 Apple Inc. All Rights Reserved. 31 Document Revision HistoryApple Inc. © 2012 Apple Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrievalsystem, or transmitted, in any form or by any means, mechanical, electronic, photocopying, recording, or otherwise, without prior written permission of Apple Inc., with the following exceptions: Any person is hereby authorized to store documentation on a single computer for personal use only and to print copies of documentation for personal use provided that the documentation contains Apple’s copyright notice. No licenses, express or implied, are granted with respect to any of the technology described in this document. Apple retains all intellectual property rights associated with the technology described in this document. This document is intended to assist application developers to develop applications only for Apple-labeled computers. Apple Inc. 1 Infinite Loop Cupertino, CA 95014 408-996-1010 Apple, the Apple logo, Cocoa, Mac, OS X, Tiger, and Xcode are trademarks of Apple Inc., registered in the U.S. and other countries. iCloud is a service mark of Apple Inc., registered in the U.S. and other countries. iOS is a trademark or registered trademark of Cisco in the U.S. and other countries and is used under license. Even though Apple has reviewed this document, APPLE MAKES NO WARRANTY OR REPRESENTATION, EITHER EXPRESS OR IMPLIED, WITH RESPECT TO THIS DOCUMENT, ITS QUALITY, ACCURACY, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.ASARESULT, THISDOCUMENT IS PROVIDED “AS IS,” AND YOU, THE READER, ARE ASSUMING THE ENTIRE RISK AS TO ITS QUALITY AND ACCURACY. IN NO EVENT WILL APPLE BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL,OR CONSEQUENTIAL DAMAGES RESULTING FROM ANY DEFECT OR INACCURACY IN THIS DOCUMENT, even if advised of the possibility of such damages. THE WARRANTY AND REMEDIES SET FORTH ABOVE ARE EXCLUSIVE AND IN LIEU OF ALL OTHERS, ORAL OR WRITTEN, EXPRESS OR IMPLIED. No Apple dealer, agent, or employee is authorized to make any modification, extension, or addition to this warranty. Some states do not allow the exclusion or limitation of implied warranties or liability for incidental or consequential damages, so the above limitation or exclusion may not apply to you. This warranty gives you specific legal rights, and you may also have other rights which vary from state to state. Kernel Programming GuideContents About This Document 9 Who Should Read This Document 9 Road Map 9 Other Apple Publications 11 Mach API Reference 11 Information on the Web 12 Keep Out 13 Why You Should Avoid Programming in the Kernel 13 Kernel Architecture Overview 14 Darwin 15 Architecture 16 Mach 17 BSD 18 I/O Kit 19 Kernel Extensions 19 The Early Boot Process 21 Boot ROM 21 The Boot Loader 21 Rooting 22 Security Considerations 24 Security Implications of Paging 25 Buffer Overflows and Invalid Input 26 User Credentials 27 Basic User Credentials 28 Access Control Lists 29 Remote Authentication 29 One-Time Pads 30 Time-based authentication 30 Temporary Files 31 /dev/mem and /dev/kmem 31 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 2Key-based Authentication and Encryption 32 Public Key Weaknesses 33 Using Public Keys for Message Exchange 35 Using Public Keys for Identity Verification 35 Using Public Keys for Data Integrity Checking 35 Encryption Summary 36 Console Debugging 36 Code Passing 37 Performance Considerations 39 Interrupt Latency 39 Locking Bottlenecks 40 Working With Highly Contended Locks 40 Reducing Contention by Decreasing Granularity 41 Code Profiling 42 Using Counters for Code Profiling 42 Lock Profiling 43 Kernel Programming Style 45 C++ Naming Conventions 45 Basic Conventions 45 Additional Guidelines 46 Standard C Naming Conventions 47 Commonly Used Functions 48 Performance and Stability Tips 50 Performance and Stability Tips 50 Stability Tips 52 Style Summary 52 Mach Overview 53 Mach Kernel Abstractions 53 Tasks and Threads 54 Ports, Port Rights, Port Sets, and Port Namespaces 55 Memory Management 57 Interprocess Communication (IPC) 58 IPC Transactions and Event Dispatching 59 Message Queues 59 Semaphores 59 Notifications 60 Locks 60 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 3 ContentsRemote Procedure Call (RPC) Objects 60 Time Management 60 Memory and Virtual Memory 61 OS X VM Overview 61 Memory Maps Explained 63 Named Entries 64 Universal Page Lists (UPLs) 65 Using Mach Memory Maps 66 Other VM and VM-Related Subsystems 68 Pagers 68 Working Set Detection Subsystem 69 VM Shared Memory Server Subsystem 69 Address Spaces 70 Background Info on PCI Address Translation 70 IOMemoryDescriptor Changes 71 VM System and pmap Changes: 72 Kernel Dependency Changes 72 Summary 72 Allocating Memory in the Kernel 73 Allocating Memory From a Non-I/O-Kit Kernel Extension 73 Allocating Memory From the I/O Kit 74 Allocating Memory In the Kernel Itself 75 Mach Scheduling and Thread Interfaces 77 Overview of Scheduling 77 Why Did My Thread Priority Change? 78 Using Mach Scheduling From User Applications 79 Using the pthreads API to Influence Scheduling 79 Using the Mach Thread API to Influence Scheduling 80 Using the Mach Task API to Influence Scheduling 83 Kernel Thread APIs 85 Creating and Destroying Kernel Threads 85 SPL and Friends 86 Wait Queues and Wait Primitives 87 Bootstrap Contexts 91 How Contexts Affect Users 92 How Contexts Affect Developers 93 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 4 ContentsI/O Kit Overview 94 Redesigning the I/O Model 94 I/O Kit Architecture 96 Families 96 Drivers 97 Nubs 97 Connection Example 98 For More Information 100 BSD Overview 101 BSD Facilities 102 Differences between OS X and BSD 103 For Further Reading 104 File Systems Overview 106 Working With the File System 106 VFS Transition 107 Network Architecture 108 Boundary Crossings 109 Security Considerations 110 Choosing a Boundary Crossing Method 110 Kernel Subsystems 111 Bandwidth and Latency 111 Mach Messaging and Mach Interprocess Communication (IPC) 112 Using Well-Defined Ports 113 Remote Procedure Calls (RPC) 113 Calling RPC From User Applications 116 BSD syscall API 116 BSD ioctl API 116 BSD sysctl API 117 General Information on Adding a sysctl 118 Adding a sysctl Procedure Call 118 Registering a New Top Level sysctl 121 Adding a Simple sysctl 122 Calling a sysctl From User Space 123 Memory Mapping and Block Copying 125 Summary 127 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 5 ContentsSynchronization Primitives 128 Semaphores 128 Condition Variables 130 Locks 132 Spinlocks 132 Mutexes 134 Read-Write Locks 136 Spin/Sleep Locks 138 Using Lock Functions 139 Miscellaneous Kernel Services 142 Using Kernel Time Abstractions 142 Obtaining Time Information 142 Event and Timer Waits 143 Handling Version Dependencies 145 Boot Option Handling 146 Queues 147 Installing Shutdown Hooks 148 Kernel Extension Overview 150 Implementation of a Kernel Extension (KEXT) 151 Kernel Extension Dependencies 151 Building and Testing Your Extension 152 Debugging Your KEXT 153 Installed KEXTs 154 Building and Debugging Kernels 155 Adding New Files or Modules 155 Modifying the Configuration Files 155 Modifying the Source Code Files 157 Building Your First Kernel 158 Building an Alternate Kernel Configuration 160 When Things Go Wrong: Debugging the Kernel 161 Setting Debug Flags in Open Firmware 161 Avoiding Watchdog Timer Problems 163 Choosing a Debugger 164 Using gdb for Kernel Debugging 164 Using ddb for Kernel Debugging 169 Bibliography 175 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 6 ContentsApple OS X Publications 175 General UNIX and Open Source Resources 175 BSD and UNIX Internals 176 Mach 177 Networking 178 Operating Systems 179 POSIX 179 Programming 179 Websites and Online Resources 180 Security and Cryptography 181 Document Revision History 182 Glossary 184 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 7 ContentsFigures, Tables, and Listings Kernel Architecture Overview 14 Figure 3-1 OS X architecture 14 Figure 3-2 Darwin and OS X 15 Figure 3-3 OS X kernel architecture 16 Kernel Programming Style 45 Table 7-1 Commonly used C functions 49 Mach Scheduling and Thread Interfaces 77 Table 10-1 Thread priority bands 77 Table 10-2 Thread policies 81 Table 10-3 Task roles 83 I/O Kit Overview 94 Figure 12-1 I/O Kit architecture 98 Synchronization Primitives 128 Listing 17-1 Allocating lock attributes and groups (lifted liberally from kern_time.c) 139 Building and Debugging Kernels 155 Table 20-1 Debugging flags 163 Table 20-2 Switch options in ddb 171 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 8The purpose of this document is to provide fundamental high-level information about the OS X core operating-system architecture. It also provides background for system programmers and developers of device drivers, file systems, and network extensions. In addition, it goes into detail about topics of interest to kernel programmers as a whole. This is not a document on drivers. It covers device drivers at a high level only. It does, however, cover some areas of interest to driver writers, such as crossing the user-kernel boundary. If you are writing device drivers, you should primarily read the document I/O Kit Fundamentals, but you may still find this document helpful as background reading. Who Should Read This Document This document has a wide and diverse audience—specifically, the set of potential system software developers for OS X, including the following sorts of developers: ● device-driver writers ● network-extension writers ● file-system writers ● developers of software that modifies file system data on-the-fly ● system programmers familiar with BSD, Linux, and similar operating systems ● developers who want to learn about kernel programming If you fall into one of these categories, you may find this document helpful. It is important to stress the care needed when writing code that resides in the kernel, however, as noted in “Keep Out” (page 13). Road Map The goal of this document is to describe the various major components of OS X at a conceptual level, then provide more detailed programming information for developers working in each major area. It is divided into several parts. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 9 About This DocumentThe first part is a kernel programming overview, which discusses programming guidelines that apply to all aspects of kernel programming. This includes issues such as security, SMP safety, style, performance, and the OS X kernel architecture as a whole. This part contains the chapters “Keep Out” (page 13), “Kernel Architecture Overview” (page 14), “The Early Boot Process” (page 21), “Security Considerations” (page 24), “Performance Considerations” (page 39), and “Kernel Programming Style” (page 45). The next part describes Mach and the bootstrap task, including information about IPC, bootstrap contexts, ports and port rights, and so on. This includes the chapters “Mach Overview” (page 53), “Memory and Virtual Memory” (page 61), “Mach Scheduling and Thread Interfaces” (page 77), and “Bootstrap Contexts” (page 91). The third part describes the I/O Kit and BSD. The I/O Kit is described at only a high level, since it is primarily of interest to driver developers. The BSD subsystem is covered in more detail, including descriptions of BSD networking and file systems. This includes the chapters “I/O Kit Overview” (page 94), “BSD Overview” (page 101), “File Systems Overview” (page 106), and “Network Architecture” (page 108). The fourth part describes kernelservices, including boundary crossings,synchronization, queues, clocks, timers, shutdown hooks, and boot option handling. This includes the chapters “Boundary Crossings” (page 109), “Synchronization Primitives” (page 128), and “Miscellaneous Kernel Services” (page 142). The fifth part explains how to build and debug the kernel and kernel extensions. This includes the chapters “Kernel Extension Overview” (page 150) and “Building and Debugging Kernels” (page 155). Each part begins with an overview chapter or chapters, followed by chapters that address particular areas of interest. The document ends with a glossary of terms used throughout the preceding chapters as well as a bibliography which provides numerous pointers to other reference materials. Glossary terms are highlighted in bold when first used. While most terms are defined when they first appear, the definitions are all in the glossary for convenience. If a term seems familiar, it probably means what you think it does. If it’s unfamiliar, check the glossary. In any case, all readers may want to skim through the glossary, in case there are subtle differences between OS X usage and that of other operating systems. The goal of this document is very broad, providing a firm grounding in the fundamentals of OS X kernel programming for developers from many backgrounds. Due to the complex nature of kernel programming and limitations on the length of this document, however, it is not always possible to provide introductory material for developers who do not have at least some background in their area of interest. It is also not possible to cover every detail of certain parts of the kernel. If you run into problems, you should join the appropriate Darwin discussion list and ask questions. You can find the lists at http://www.lists.apple.com/. For this reason, the bibliography contains high-level references that should help familiarize you with some of the basic concepts that you need to understand fully the material in this document. About This Document Road Map 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 10This document is, to a degree, a reference document. The introductory sections should be easily read, and we recommend that you do so in order to gain a general understanding of each topic. Likewise, the first part of each chapter, and in many cases, of sections within chapters, will be tailored to providing a general understanding of individual topics. However, you should not plan to read this document cover to cover, but rather, take note of topics of interest so that you can refer back to them when the need arises. Other Apple Publications This document, Kernel Programming , is part of the Apple Reference Library. Be sure to read the first document in the series, Mac Technology Overview, if you are not familiar with OS X. You can obtain other documents from the Apple Developer Documentation website at http://developer.apple.com/documentation. Mach API Reference If you plan to do extensive work inside the OS X kernel, you may find it convenient to have a complete Mach API reference, since this document only documents the most common and useful portions of the Mach API. In order to better understand certain interfaces, it may also be helpful to study the implementations that led up to those used in OS X, particularly to fill in gaps in understanding of the fundamental principles of the implementation. OS X is based on the Mach 3.0 microkernel, designed by Carnegie Mellon University, and later adapted to the Power Macintosh by Apple and the Open Software Foundation Research Institute (now part of Silicomp). This was known as osfmk, and was part of MkLinux (http://www.mklinux.org). Later, this and code from OSF’s commercial development efforts were incorporated into Darwin’s kernel. Throughout this evolutionary process, the Mach APIs used in OS X diverged in many ways from the original CMU Mach 3 APIs. You may find older versions of the Mach source code interesting, both to satisfy historical curiosity and to avoid remaking mistakes made in earlier implementations. MkLinux maintains an active CVS repository with their recent versions of Mach kernel source code. Older versions can be obtained through various Internet sites. You can also find CMU Mach white papers by searching for Mach on the CMU computer science department’s website (http://www.cs.cmu.edu), along with various source code samples. Up-to-date versions of the Mach 3 APIsthat OS X provides are described in the Mach API reference in the kernel sources. The kernel sources can be found in the xnu project on http://kernel.macosforge.org/. About This Document Other Apple Publications 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 11Information on the Web Apple maintains several websites where developers can go for general and technical information on OS X. ● Apple Developer Connection: Developer Documentation (http://developer.apple.com/documentation). Features the same documentation that is installed on OS X, except that often the documentation is more up-to-date. Also includes legacy documentation. ● Apple Developer Connection: OS X (http://developer.apple.com/devcenter/mac/). Offers SDKs, release notes, product notes and news, and other resources and information related to OS X. ● AppleCare Tech Info Library (http://www.apple.com/support/). Contains technical articles, tutorials, FAQs, technical notes, and other information. About This Document Information on the Web 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 12This document assumes a broad general understanding of kernel programming concepts. There are many good introductory operating systems texts. This is not one of them. For more information on basic operating systems programming, you should consider the texts mentioned in the bibliography at the end of this document. Many developers are justifiably cautious about programming in the kernel. A decision to program in the kernel is not to be taken lightly. Kernel programmers have a responsibility to users that greatly surpasses that of programmers who write user programs. Why You Should Avoid Programming in the Kernel Kernel code must be nearly perfect. A bug in the kernel could cause random crashes, data corruption, or even render the operating system inoperable. It is even possible for certain errant operations to cause permanent and irreparable damage to hardware, for example, by disabling the cooling fan and running the CPU full tilt. Kernel programming is a black art that should be avoided if at all possible. Fortunately, kernel programming is usually unnecessary. You can write most software entirely in user space. Even most device drivers (FireWire and USB, for example) can be written as applications, rather than as kernel code. A few low-level drivers must be resident in the kernel's address space, however, and this document might be marginally useful if you are writing drivers that fall into this category. Despite parts of this document being useful in driver writing, this is not a document about writing drivers. In OS X, you write device drivers using the I/O Kit. While this document covers the I/O Kit at a conceptual level, the details of I/O Kit programming are beyond the scope of this document. Driver writers are encouraged to read I/O Kit Fundamentals for detailed information about the I/O Kit. This document covers most aspects of kernel programmingwith the exception of device drivers. Covered topics include scheduling, virtual memory pagers and policies, Mach IPC, file systems, networking protocol stacks, process and thread management, kernel security, synchronization, and a number of more esoteric topics. To summarize, kernel programming is an immense responsibility. You must be exceptionally careful to ensure that your code does not cause the system to crash, does not provide any unauthorized user accessto someone else’s files or memory, does not introduce remote or local root exploits, and does not cause inadvertent data loss or corruption. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 13 Keep OutOS X provides many benefits to the Macintosh user and developer communities. These benefits include improved reliability and performance, enhanced networking features, an object-based system programming interface, and increased support for industry standards. In creatingOS X, Apple has completely re-engineered the MacOS core operating system. Forming the foundation of OS X is the kernel. Figure 3-1 (page 14) illustrates the OS X architecture. Figure 3-1 OS X architecture Carbon Cocoa BSD Java (JDK) Classic BSD Core Services Kernel environment Application Services QuickTime Application environment The kernel provides many enhancements for OS X. These include preemption, memory protection, enhanced performance, improved networking facilities, support for both Macintosh (Extended and Standard) and non-Macintosh (UFS, ISO 9660, and so on) file systems, object-oriented APIs, and more. Two of these features, preemption and memory protection, lead to a more robust environment. In Mac OS 9, applications cooperate to share processor time. Similarly, all applications share the memory of the computer among them. Mac OS 9 is a cooperative multitasking environment. The responsiveness of all processes is compromised if even a single application doesn’t cooperate. On the other hand, real-time applications such as multimedia need to be assured of predictable, time-critical, behavior. In contrast, OS X is a preemptive multitasking environment. In OS X, the kernel provides enforcement of cooperation,scheduling processesto share time (preemption). Thissupportsreal-time behavior in applications that require it. In OS X, processes do not normally share memory. Instead, the kernel assigns each process its own address space, controlling access to these address spaces. This control ensures that no application can inadvertently access or modify another application’s memory (protection). Size is not an issue; with the virtual memory system included in OS X, each application has access to its own 4 GB address space. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 14 Kernel Architecture OverviewViewed together, all applications are said to run in user space, but this does not imply that they share memory. User space is simply a term for the combined address spaces of all user-level applications. The kernel itself has its own address space, called kernel space. In OS X, no application can directly modify the memory of the system software (the kernel). Although user processes do not share memory by default as in Mac OS 9, communication (and even memory sharing) between applications is still possible. For example, the kernel offers a rich set of primitives to permit some sharing of information among processes. These primitives include shared libraries, frameworks, and POSIX shared memory. Mach messaging provides another approach, handing memory from one process to another. Unlike Mac OS 9, however, memory sharing cannot occur without explicit action by the programmer. Darwin The OS X kernel is an Open Source project. The kernel, along with other core parts of OS X are collectively referred to as Darwin. Darwin is a complete operating system based on many of the same technologies that underlie OS X. However, Darwin does not include Apple’s proprietary graphics or applications layers, such as Quartz, QuickTime, Cocoa, Carbon, or OpenGL. Figure 3-2 (page 15) shows the relationship between Darwin and OS X. Both build upon the same kernel, but OS X adds Core Services, Application Services and QuickTime, as well as the Classic, Carbon, Cocoa, and Java (JDK) application environments. Both Darwin and OS X include the BSD command-line application environment; however, in OS X, use of environment is not required, and thus it is hidden from the user unless they choose to access it. Figure 3-2 Darwin and OS X Carbon Cocoa BSD Java (JDK) Classic BSD Core Services Kernel environment Application Services QuickTime Application environment Darwin technology is based on BSD, Mach 3.0, and Apple technologies. Best of all, Darwin technology is Open Source technology, which meansthat developers have full accessto the source code. In effect, OS X third-party developers can be part of the Darwin core system software development team. Developers can also see how Apple is doing thingsin the core operating system and adopt (or adapt) code to use within their own products. Refer to the Apple Public Source License (APSL) for details. Kernel Architecture Overview Darwin 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 15Because the same software forms the core of both OS X and Darwin, developers can create low-level software that runs on both OS X and Darwin with few, if any, changes. The only difference is likely to be in the way the software interacts with the application environment. Darwin is based on proven technology from many sources. A large portion of this technology is derived from FreeBSD, a version of 4.4BSD that offers advanced networking, performance,security, and compatibility features. Other parts of the system software, such as Mach, are based on technology previously used in Apple’s MkLinux project, in OS X Server, and in technology acquired from NeXT. Much of the code is platform-independent. All of the core operating-system code is available in source form. The core technologies have been chosen for several reasons. Mach provides a clean set of abstractions for dealing with memory management, interprocess(and interprocessor) communication (IPC), and other low-level operating-system functions. In today’s rapidly changing hardware environment, this provides a useful layer of insulation between the operating system and the underlying hardware. BSD is a carefully engineered, mature operating system with many capabilities. In fact, most of today’s commercial UNIX and UNIX-like operating systems contain a great deal of BSD code. BSD also provides a set of industry-standard APIs. New technologies,such asthe I/OKit and Network Kernel Extensions(NKEs), have been designed and engineered by Apple to take advantage of advanced capabilities,such asthose provided by an object-oriented programming model. OS X combines these new technologies with time-tested industry standards to create an operating system that is stable, reliable, flexible, and extensible. Architecture The foundation layer of Darwin and OS X is composed of several architectural components, as shown in Figure 3-3 (page 16). Taken together, these components form the kernel environment. Figure 3-3 OS X kernel architecture Common services Kernel environment Application environments Mach File system BSD Networking NKE Drivers I/O Kit Kernel Architecture Overview Architecture 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 16Important: Note that OS X uses the term kernel somewhat differently than you might expect. “A kernel, in traditional operating-system terminology, is a small nucleus of software that provides only the minimal facilities necessary for implementing additional operating-system services.” — from The Design and Implementation of the 4.4 BSD Operating System, McKusick, Bostic, Karels, and Quarterman, 1996. Similarly, in traditional Mach-based operating systems, the kernel refers to the Mach microkernel and ignores additional low-level code without which Mach does very little. In OS X, however, the kernel environment contains much more than the Mach kernel itself. The OS X kernel environment includes the Mach kernel, BSD, the I/O Kit, file systems, and networking components. These are often referred to collectively as the kernel. Each of these components is described briefly in the following sections. For further details, refer to the specific component chapters or to the reference material listed in the bibliography. Because OS X contains three basic components (Mach, BSD, and the I/O Kit), there are also frequently as many as three APIs for certain key operations. In general, the API chosen should match the part of the kernel where it is being used, which in turn is dictated by what your code is attempting to do. The remainder of this chapter describes Mach, BSD, and the I/O Kit and outlines the functionality that is provided by those components. Mach Mach manages processor resources such as CPU usage and memory, handles scheduling, provides memory protection, and provides a messaging-centered infrastructure to the rest of the operating-system layers. The Mach component provides ● untyped interprocess communication (IPC) ● remote procedure calls (RPC) ● scheduler support for symmetric multiprocessing (SMP) ● support for real-time services ● virtual memory support ● support for pagers ● modular architecture General information about Mach may be found in the chapter “Mach Overview” (page 53). Information about scheduling can be found in the chapter “Mach Scheduling and Thread Interfaces” (page 77). Information about the VM system can be found in “Memory and Virtual Memory” (page 61). Kernel Architecture Overview Architecture 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 17BSD Above the Mach layer, the BSD layer provides “OS personality” APIs and services. The BSD layer is based on the BSD kernel, primarily FreeBSD. The BSD component provides ● file systems ● networking (except for the hardware device level) ● UNIX security model ● syscall support ● the BSD process model, including process IDs and signals ● FreeBSD kernel APIs ● many of the POSIX APIs ● kernel support for pthreads (POSIX threads) The BSD component is described in more detail in the chapter “BSD Overview” (page 101). Networking OS X networking takes advantage of BSD’s advanced networking capabilities to provide support for modern features, such as Network Address Translation (NAT) and firewalls. The networking component provides ● 4.4BSD TCP/IP stack and socket APIs ● support for both IP and DDP (AppleTalk transport) ● multihoming ● routing ● multicast support ● server tuning ● packet filtering ● Mac OS Classic support (through filters) More information about networking may be found in the chapter “Network Architecture” (page 108). Kernel Architecture Overview Architecture 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 18File Systems OS X providessupport for numeroustypes of file systems, including HFS, HFS+, UFS, NFS, ISO 9660, and others. The default file-system type is HFS+; OS X boots (and “roots”) from HFS+, UFS, ISO, NFS, and UDF. Advanced features of OS X file systems include an enhanced Virtual File System (VFS) design. VFS provides for a layered architecture (file systems are stackable). The file system component provides ● UTF-8 (Unicode) support ● increased performance over previous versions of Mac OS. More information may be found in the chapter “File Systems Overview” (page 106). I/O Kit The I/O Kit provides a framework forsimplified driver development,supporting many categories of devices.The I/O Kit features an object-oriented I/O architecture implemented in a restricted subset of C++. The I/O Kit framework is both modular and extensible. The I/O Kit component provides ● true plug and play ● dynamic device management ● dynamic (“on-demand”) loading of drivers ● power management for desktop systems as well as portables ● multiprocessor capabilities The I/O Kit is described in greater detail in the chapter “I/O Kit Overview” (page 94). Kernel Extensions OS X provides a kernel extension mechanism as a means of allowing dynamic loading of pieces of code into kernel space, without the need to recompile. These pieces of code are known generically as plug-ins or, in the OS X kernel environment, as kernel extensions or KEXTs. Because KEXTs provide both modularity and dynamic loadability, they are a natural choice for any relatively self-contained service that requires access to interfaces that are not exported to user space. Many of the components of the kernel environment support this extension mechanism, though they do so in different ways. For example, some of the new networking features involve the use of network kernel extensions (NKEs). These are discussed in the chapter “Network Architecture” (page 108). Kernel Architecture Overview Kernel Extensions 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 19The ability to dynamically add a new file-system implementation is based on VFS KEXTs. Device drivers and device familiesin the I/O Kit are implemented using KEXTs. KEXTs make development much easier for developers writing drivers or those writing code to support a new volume format or networking protocol. KEXTs are discussed in more detail in the chapter “Kernel Extension Overview” (page 150). Kernel Architecture Overview Kernel Extensions 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 20Boot ROM When the power to a Macintosh computer is turned on, the BootROM firmware is activated. BootROM (which is part of the computer’s hardware) hastwo primary responsibilities: it initializessystem hardware and itselects an operating system to run. BootROM has two components to help it carry out these functions: ● POST (Power-On Self Test) initializes some hardware interfaces and verifies that sufficient memory is available and in a good state. ● EFI does basic hardware initialization and selects which operating system to use. If multiple installations of OS X are available, BootROM chooses the one that was last selected by the Startup Disk System Preference. The user can override this choice by holding down the Option key while the computer boots, which causes EFI to display a screen for choosing the boot volume. The Boot Loader Once BootROM is finished and an OS X partition has been selected, control passes to the boot.efi boot loader. The principal job of this boot loader is to load the kernel environment. As it does this, the boot loader draws the “booting” image on the screen. If full-disk encryption is enabled, the boot loader is responsible for drawing the login UI and prompting for the user’s password, which needed to accessthe encrypted disk to boot from it. (This UI is drawn by loginwindow otherwise.) In the simplest case, the boot loader can be found in the /System/Library/CoreServices directory on the root partition, in a file named boot.efi. Note: Booting from a UFS volume is deprecated as of OS X v10.5. In order to speed up boot time, the boot loader uses several caches. The contents and location of these caches varies between versions of OS X, but knowing some details about the caching may be helpful when debugging kernel extensions. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 21 The Early Boot ProcessAfter you install or modify a kernel extension, touch the /System/Library/Extensions directory; the system rebuilds the caches automatically. Important: You should not depend on the implementation details of the kernel caches in your software. In OS X v10.7, the boot loader looks for the unified prelinked kernel. This cache contains all kernel extensions that may be needed to boot a Mac with any hardware configuration, with the extensions already linked against the kernel. It islocated at /System/Library/Caches/com.apple.kext.caches/Startup/kernelcache. In OS X v10.6 and earlier, the boot loader first looks for the prelinked kernel (also called the kernel cache). This cache contains exactly the set of kernel extensions that were needed during the previous system startup, already linked against the kernel. If the prelinked kernel is missing or unusable (for example, because a hardware configuration has changed), the booter looks for the mkext cache, which contains all kernel extensions that may be needed to boot the system. Using the mkext cache is much slower because the linker must be run. On OS X v10.5 and v10.6, these caches are located in /System/Library/Caches/com.apple.kext.caches/Startup/; on previous versions of OS X, it was located at /System/Library/Caches/com.apple.kernelcaches/. Finally, if the caches cannot be used, the boot loader searches /System/Library/Extensions for drivers and other kernel extensions whose OSBundleRequired property is set to a value appropriate to the type of boot (for example, local or network boot). This process is very slow, because the Info.plist file of every kernel extension must be parsed, and then the linker must be run. For more information on how drivers are loaded, see I/O Kit Fundamentals, the manual page for kextcache, and Kernel Extension Programming Topics. Rooting Once the kernel and all drivers necessary for booting are loaded, the boot loaderstartsthe kernel’sinitialization procedure. At this point, enough drivers are loaded for the kernel to find the root device. The kernel initializes the Mach and BSD data structures and then initializes the I/O Kit. The I/O Kit links the loaded drivers into the kernel, using the device tree to determine which drivers to link. Once the kernel finds the root device, it roots(*) BSD off of it. The Early Boot Process Rooting 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 22Note: As a terminology aside, the term “boot” was historically reserved for loading a bootstrap loader and kernel off of a disk or partition. In more recent years, the usage has evolved to allow a second meaning: the entire process from initial bootstrap until the OS is generally usable by an end user. In this case, the term is used according to the former meaning. As used here, the term “root” refersto mounting a partition asthe root, or top-level, filesystem. Thus, while the OS boots off of the root partition, the kernel rootsthe OS off of the partition before executing startup scripts from it. Boot≠Root is a technology that allows the system to boot from a partition other than the root partition. This is used to boot systems where the root partition is encrypted using full-disk encryption, or where the root partition islocated on a device which requires additional drivers(such as a RAID array). Boot≠Root uses a helper partition to store the files needed to boot, such as the kernel cache. For more information on how to set up the property in a filter-scheme driver,see “Developing a Filter Scheme” in Mass StorageDeviceDriver Programming Guide . The Early Boot Process Rooting 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 23Kernel-level security can mean many things, depending on what kind of kernel code you are writing. This chapter points out some common security issues at the kernel or near-kernel level and where applicable, describes ways to avoid them. These issues are covered in the following sections: ● “Security Implications of Paging” (page 25) ● “Buffer Overflows and Invalid Input” (page 26) ● “User Credentials” (page 27) ● “Remote Authentication” (page 29) ● “Temporary Files” (page 31) ● “/dev/mem and /dev/kmem” (page 31) ● “Key-based Authentication and Encryption” (page 32) ● “Console Debugging” (page 36) ● “Code Passing” (page 37) Many of these issues are also relevant for application programming, but are crucial for programmers working in the kernel. Others are special considerations that application programers might not expect or anticipate. Note: The terms cleartext and plaintext both refer to unencrypted text. These terms can generally be used interchangeably, although in some circles, the term cleartext is restricted to unencrypted transmission across a network. However, in other circles, the term plaintext (orsometimes plain text) refers to plain ASCII text (as opposed to HTML or rich text. To avoid any potential confusion, this chapter will use the term cleartext to refer to unencrypted text. In order to understand security in OS X, it is important to understand that there are two security models at work. One of these is the kernel security model, which is based on users, groups, and very basic per-user and per-group rights, which are, in turn, coupled with access control lists for increased flexibility. The other is a user-level security model, which is based on keys, keychains, groups, users, password-based authentication, and a host of other details that are beyond the scope of this document. The user level of security contains two basic features that you should be aware of as a kernel programmer: Security Server and Keychain Manager. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 24 Security ConsiderationsThe Security Server consists of a daemon and various accesslibrariesfor caching permission to do certain tasks, based upon various means of authentication, including passwords and group membership. When a program requests permission to do something, the Security Server basically says “yes” or “no,” and caches that decision so that further requestsfrom that user (forsimilar actions within a single context) do not require reauthentication for a period of time. The Keychain Manager is a daemon that provides services related to the keychain, a central repository for a user’s encryption/authentication keys. For more high level information on keys,see “Key-based Authentication and Encryption” (page 32). The details of the user-level security model use are far beyond the scope of this document. However, if you are writing an application that requires services of this nature, you should consider taking advantage of the Security Server and Keychain Manager from the user-space portion of your application, rather than attempting equivalent services in the kernel. More information about these services can be found in Apple’s Developer Documentation website at http://developer.apple.com/documentation. Security Implications of Paging Paging has long been a major problem for security-conscious programmers. If you are writing a program that does encryption, the existence of even a small portion of the cleartext of a document in a backing store could be enough to reduce the complexity of breaking that encryption by orders of magnitude. Indeed, many types of data,such as hashes, unencrypted versions ofsensitive data, and authentication tokens, should generally not be written to disk due to the potential for abuse. This raises an interesting problem. There is no good way to deal with this in user space (unless a program is running as root). However, for kernel code, it is possible to prevent pages from being written out to a backing store. This process is referred to as “wiring down” memory, and is described further in “Memory Mapping and Block Copying” (page 125). The primary purpose of wired memory is to allow DMA-based I/O. Since hardware DMA controllers generally do not understand virtual addressing, information used in I/O must be physically in memory at a particular location and must not move until the I/O operation is complete. This mechanism can also be used to prevent sensitive data from being written to a backing store. Because wired memory can never be paged out (until it is unwired), wiring large amounts of memory has drastic performance repercussions, particularly on systems with small amounts of memory. For this reason, you should take care not to wire down memory indiscriminately and only wire down memory if you have a very good reason to do so. In OS X, you can wire down memory at allocation time or afterwards. To wire memory at allocation time: ● In I/O Kit, call IOMalloc and IOFree to allocate and free wired memory. Security Considerations Security Implications of Paging 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 25● In other kernel extensions, call OSMalloc and OSFree and pass a tag type whose flags are set to OSMT_DEFAULT. ● In user space code, allocate page-sized quantities with your choice of API, and then call mlock(2) to wire them. ● Inside the kernel itself (not in kernel extensions), you can also use kmem_alloc and related functions. For more information on wired memory, see “Memory Mapping and Block Copying” (page 125). Buffer Overflows and Invalid Input Buffer overflows are one of the more common bugs in both application and kernel programming. The most common cause is failing to allocate space for the NULL character that terminates a string in C or C++. However, user input can also cause buffer overflows if fixed-size input buffers are used and appropriate care is not taken to prevent overflowing these buffers. The most obvious protection, in this case, is the best one. Either don’t use fixed-length buffers or add code to reject or truncate input that overflows the buffer. The implementation details in either case depend on the type of code you are writing. For example, if you are working with strings and truncation is acceptable, instead of using strcpy, you should use strlcpy to limit the amount of data to copy. OS X provides length-limited versions of a number of string functions, including strlcpy, strlcat, strncmp, snprintf, and vsnprintf. If truncation of data is not acceptable, you must explicitly call strlen to determine the length of the input string and return an error if it exceeds the maximum length (one less than the buffer size). Other types of invalid input can be somewhat harder to handle, however. As a general rule, you should be certain that switch statements have a default case unless you have listed every legal value for the width of the type. A common mistake is assuming that listing every possible value of an enum type provides protection. An enum is generally implemented as either a char or an int internally. A careless or malicious programmer could easily pass any value to a kernel function, including those not explicitly listed in the type, simply by using a different prototype that defines the parameter as, for example, an int. Another common mistake is to assume that you can dereference a pointer passed to your function by another function. You should always check for null pointers before dereferencing them. Starting a function with int do_something(bufptr *bp, int flags) { Security Considerations Buffer Overflows and Invalid Input 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 26char *token = bp->b_data; isthe surest way to guarantee thatsomeone else will passin a null buffer pointer, either maliciously or because of programmer error. In a user program, this is annoying. In a file system, it is devastating. Security is particularly important for kernel code that draws input from a network. Assumptions about packet size are frequently the cause of security problems. Always watch for packets that are too big and handle them in a reasonable way. Likewise, always verify checksums on packets. This can help you determine if a packet was modified, damaged, or truncated in transit, though it is far from foolproof. If the validity of data from a network is of vital importance, you should use remote authentication, signing, and encryption mechanisms such as those described in “Remote Authentication” (page 29) and “Key-based Authentication and Encryption” (page 32). User Credentials As described in the introduction to this chapter, OS X has two different means of authenticating users. The user-levelsecurity model (including the Keychain Manager and the Security Server) is beyond the scope of this document. The kernel security model, however, is of greater interest to kernel developers, and is much more straightforward than the user-level model. The kernel security model is based on two mechanisms: basic user credentials and ACL permissions. The first, basic user credentials, are passed around within the kernel to identify the current user and group of the calling process. The second authentication mechanism, access control lists (ACLs), provides access control at a finer level of granularity. One of the most important things to remember when working with credentials is that they are per process, not per context. This is important because a process may not be running as the console user. Two examples of this are processes started from an ssh session (since ssh runs in the startup context) and setuid programs (which run as a different user in the same login context). It is crucial to be aware of these issues. If you are communicating with a setuid root GUI application in a user’s login context, and if you are executing another application or are reading sensitive data, you probably want to treat it as if it had the same authority as the console user, not the authority of the effective user ID caused by running setuid. This is particularly problematic when dealing with programs that run as setuid root if the console user is not in the admin group. Failure to perform reasonable checks can lead to major security holes down the road. However, this is not a hard and fast rule. Sometimes it is not obvious whether to use the credentials of the running process or those of the console user. In such cases, it is often reasonable to have a helper application show a dialog box on the console to require interaction from the console user. If this is not possible, a good Security Considerations User Credentials 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 27rule of thumb is to assume the lesser of the privileges of the current and console users, as it is almost always better to have kernel code occasionally fail to provide a needed service than to provide that service unintentionally to an unauthorized user or process. It is generally easier to determine the console user from a user space application than from kernel space code. Thus, you should generally do such checks from user space. If that is not possible, however, the variable console_user (maintained by the VFS subsystem) will give you the uid of the last owner of /dev/console (maintained by a bit of code in the chown system call). Thisis certainly not an idealsolution, but it does provide the most likely identity of the console user. Since this is only a “best guess,” however, you should use this only if you cannot do appropriate checking in user space. Basic User Credentials Basic user credentials used in the kernel are stored in a variable of type struct ucred. These are mostly used in specialized parts of the kernel—generally in places where the determining factor in permissions is whether or not the caller is running as the root user. This structure has four fields: ● cr_ref—reference count (used internally) ● cr_uid—user ID ● cr_ngroups—number of groups in cr_groups ● cr_groups[NGROUPS]—list of groups to which the user belongs Thisstructure has an internal reference counter to prevent unintentionally freeing the memory associated with it while it is still in use. For this reason, you should not indiscriminately copy this object but should instead either use crdup to duplicate it or use crcopy to duplicate it and (potentially) free the original. You should be sure to crfree any copies you might make. You can also create a new, empty ucred structure with crget. The prototypes for these functions follow: ● struct ucred *crdup(struct ucred *cr) ● struct ucred *crcopy(struct ucred *cr) ● struct ucred *crget(void) ● void crfree(struct ucred *cr) Security Considerations User Credentials 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 28Note: Functions for working with basic user credential are not exported outside of the kernel, and thus are not generally available to kernel extensions. Access Control Lists Access control lists are a new feature in OS X v10.4. Access control lists are primarily used in the file system portion of the OS X kernel, and are supported through the use of the kauth API. The kauth API is described in the header file /System/Library/Frameworks/Kernel.framework/Headers/sys/kauth.h. Because this API is still evolving, detailed documentation is not yet available. Remote Authentication This is one of the more difficult problems in computer security: the ability to identify someone connecting to a computer remotely. One of the mostsecure methodsisthe use of public key cryptography, which is described in more detail in “Key-based Authentication and Encryption” (page 32). However, many other means of authentication are possible, with varying degrees of security. Some other authentication schemes include: ● blind trust ● IP-only authentication ● password (shared secret) authentication ● combination of IP and password authentication ● one-time pads (challenge-response) ● time-based authentication Most of these are obvious, and require no further explanation. However, one-time pads and time-based authentication may be unfamiliar to many people outside security circles, and are thus worth mentioning in more detail. Security Considerations Remote Authentication 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 29One-Time Pads Based on the concept of “challenge-response” pairs, one-time pad (OTP) authentication requires that both parties have an identical list of pairs of numbers, words, symbols, or whatever, sorted by the first item. When trying to access a remote system, the remote system prompts the user with a challenge. The user finds the challenge in the first column, then sends back the matching response. Alternatively, this could be an automated exchange between two pieces of software. For maximum security, no challenge should ever be issued twice. For this reason, and because these systems were initially implemented with a paper pad containing challenge-response, or CR pairs, such systems are often called one-time pads. The one-time nature of OTP authentication makesit impossible forsomeone to guessthe appropriate response to any one particular challenge by a brute force attack (by responding to that challenge repeatedly with different answers). Basically, the only way to break such a system, short of a lucky guess, is to actually know some portion of the contents of the list of pairs. For this reason, one-time pads can be used over insecure communication channels. If someone snoops the communication, they can obtain that challenge-response pair. However, that information is of no use to them, since that particular challenge will never be issued again. (It does not even reduce the potential sample space for responses, since only the challenges must be unique.) Time-based authentication Thisis probably the least understood means of authentication, though it is commonly used by such technologies as SecurID. The concept isrelatively straightforward. You begin with a mathematical function that takes a small number of parameters (two, for example) and returns a new parameter. A good example of such a function is the function that generates the set of Fibonacci numbers (possibly truncated after a certain number of bits, with arbitrary initial seed values). Take this function, and add a third parameter, t, representing time in units of k seconds. Make the function be a generating function on t, with two seed values, a and b, where f(x,y) = (x + y) MOD (2 32 ) g(t) = a, 0 t k g(t) = b, k t 2k g(t) = f (g( log k t -2),g( log k t -1)) Security Considerations Remote Authentication 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 30In other words, every k seconds, you calculate a new value based on the previous two and some equation. Then discard the oldest value, replacing it with the second oldest value, and replace the second oldest value with the value that you just generated. As long as both ends have the same notion of the current time and the original two numbers, they can then calculate the most recently generated number and use this as a shared secret. Of course, if you are writing code that does this, you should use a closed form of this equation, since calculating Fibonacci numbers recursively without additional storage grows at O(2^(t/k)), which is not practical when t is measured in years and k is a small constant measured in seconds. The security ofsuch a scheme depends on various properties of the generator function, and the details ofsuch a function are beyond the scope of this document. For more information, you should obtain an introductory text on cryptography,. such as Bruce Schneier’s Applied Cryptography . Temporary Files Temporary files are a major source of security headaches. If a program does not set permissions correctly and in the right order, this can provide a means for an attacker to arbitrarily modify or read these files. The security impact of such modifications depends on the contents of the files. Temporary files are of much less concern to kernel programmers,since most kernel code does not use temporary files. Indeed, kernel code should generally not use files at all. However, many people programming in the kernel are doing so to facilitate the use of applicationsthat may use temporary files. Assuch, thisissue is worth noting. The most common problem with temporary files is that it is often possible for a malicious third party to delete the temporary file and substitute a different one with relaxed permissions in its place. Depending on the contents of the file, this could range from being a minor inconvenience to being a relatively large security hole, particularly if the file contains a shell script that is about to be executed with the permissions of the program’s user. /dev/mem and /dev/kmem One particularly painfulsurprise to people doing security programming in most UNIX or UNIX-like environments is the existence of /dev/mem and /dev/kmem. These device files allow the root user to arbitrarily access the contents of physical memory and kernel memory, respectively. There is absolutely nothing you can do to prevent this. From a kernel perspective, root is omnipresent and omniscient. If this is a security concern for your program, then you should consider whether your program should be used on a system controlled by someone else and take the necessary precautions. Security Considerations Temporary Files 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 31Note: Support for /dev/kmem is being phased out. It is not available on Intel-based Macintosh computers in OS X v10.4. In the future, it will be removed entirely. It is not possible to write device drivers that access PCI device memory through /dev/mem in OS X. If you need to support such a driver, you must write a kernel stub driver that matches against the device and mapsits memory space into the addressspace of the user process. For more information, read about user clients in I/O Kit Fundamentals. Key-based Authentication and Encryption Key-based authentication and encryption are ostensibly some of the more secure means of authentication and encryption, and can exist in many forms. The most common forms are based upon a shared secret. The DES, 3DES (triple-DES), IDEA, twofish, and blowfish ciphers are examples of encryption schemes based on a shared secret. Passwords are an example of an authentication scheme based on a shared secret. The idea behind most key-based encryption is that you have an encryption key of some arbitrary length that is used to encode the data, and that same key is used in the opposite manner (or in some cases, in the same manner) to decode the data. The problem with shared secret security is that the initial key exchange must occur in a secure fashion. If the integrity of the key is compromised during transmission, the data integrity is lost. This is not a concern if the key can be generated ahead of time and placed at both transport endpoints in a secure fashion. However, in many cases, this is not possible or practical because the two endpoints (be they physical devices or system tasks) are controlled by different people or entities. Fortunately, an alternative exists, known as zero-knowledge proofs. The concept of a zero-knowledge proof is that two seemingly arbitrary key values, x and y, are created, and that these values are related by some mathematical function ƒ in such a way that ƒ(ƒ(a,k1),k2) = a That is, applying a well-known function to the original cleartext using the first key results in ciphertext which, when that same function is applied to the ciphertext using the second key returns the original data. This is also reversible, meaning that ƒ(ƒ(a,k2),k1) = a Security Considerations Key-based Authentication and Encryption 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 32If the function f is chosen correctly, it is extremely difficult to derive x from y and vice-versa, which would mean that there is no function that can easily transform the ciphertext back into the cleartext based upon the key used to encode it. An example of this is to choose the mathematical function to be f(a,k)=((a*k) MOD 256) + ((a*k)/256) where a is a byte of cleartext, and k is some key 8 bits in length. This is an extraordinarily weak cipher, since the function f allows you to easily determine one key from the other, but it is illustrative of the basic concept. Pick k1 to be 8 and k2 to be 32. So for a=73, (a * 8)=584. This takes two bytes, so add the bits in the high byte to the bits of the low byte, and you get 74. Repeat this process with 32. This gives you 2368. Again, add the bits from the high byte to the bits of the low byte, and you have 73 again. This mathematical concept (with very different functions), when put to practical use, is known as public key (PK) cryptography, and forms the basis for RSA and DSA encryption. Public Key Weaknesses Public key encryption can be very powerful when used properly. However, it has a number of inherent weaknesses. A complete explanation of these weaknesses is beyond the scope of this document. However, it is important that you understand these weaknesses at a high level to avoid falling into some common traps. Some commonly mentioned weakness of public key cryptography include: ● Trust model for key exchange ● Pattern sensitivity ● Short data weakness Trust Models The most commonly discussed weakness of public key cryptography is the initial key exchange process itself. If someone manages to intercept a key during the initial exchange, he or she could instead give you his or her own public key and intercept messages going to the intended party. This is known as a man-in-the-middle attack. For such services as ssh, most people either manually copy the keys from one server to another or simply assume that the initial key exchange was successful. For most purposes, this is sufficient. In particularly sensitive situations, however, this is not good enough. For this reason, there is a procedure known as key signing. There are two basic models for key signing: the central authority model and the web of trust model. Security Considerations Key-based Authentication and Encryption 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 33The central authority model is straightforward. A central certifying agency signs a given key, and says that they believe the owner of the key is who he or she claims to be. If you trust that authority, then by association, you trust keys that the authority claims are valid. The web of trust model is somewhat different. Instead of a central authority, individuals sign keys belonging to other individuals. By signing someone’s key, you are saying that you trust that the person is really who he or she claims to be and that you believe that the key really belongs to him or her. The methods you use for determining that trust will ultimately impact whether others trust your signatures to be valid. There are many different ways of determining trust, and thus many groups have their own rulesfor who should and should not sign someone else’s key. Those rules are intended to make the trust level of a key depend on the trust level of the keys that have signed it. The line between central authorities and web of trust models is not quite as clear-cut as you might think, however. Many central authorities are hierarchies of authorities, and in some cases, they are actually webs of trust among multiple authorities. Likewise, many webs of trust may include centralized repositories for keys. While those repositories don’t provide any certification of the keys, they do provide centralized access. Finally, centralized authorities can easily sign keys as part of a web of trust. There are many websites that describe webs of trust and centralized certification schemes. A good general description of several such models can be found at http://world.std.com/~cme/html/web.html. Sensitivity to Patterns and Short Messages Existing public key encryption algorithms do a good job at encrypting semi-random data. They fallshort when encrypting data with certain patterns, as these patterns can inadvertently reveal information about the keys. The particular patterns depend on the encryption scheme. Inadvertently hitting such a pattern does not allow you to determine the private key. However, they can reduce the search space needed to decode a given message. Short data weakness is closely related to pattern sensitivity. If the information you are encrypting consists of a single number, for example the number 1, you basically get a value that is closely related mathematically to the public key. If the intent is to make sure that only someone with the private key can get the original value, you have a problem. In other words, public key encryption schemes generally do not encrypt all patterns equally well. For thisreason (and because public key cryptography tendsto be slower than single key cryptography), public keys are almost never used to encrypt end-user data. Instead, they are used to encrypt a session key. This session key is then used to encrypt the actual data using a shared secret mechanism such as 3DES, AES, blowfish, and so on. Security Considerations Key-based Authentication and Encryption 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 34Using Public Keys for Message Exchange Public key cryptography can be used in many ways. When both keys are private, it can be used to send data back and forth. However this use is no more useful than a shared secret mechanism. In fact, it is frequently weaker, for the reasons mentioned earlier in the chapter. Public key cryptography becomes powerful when one key is made public. Assume that Ernie and Bert want to send coded messages. Ernie gives Bert his public key. Assuming that the key was not intercepted and replaced with someone else’s key, Bert can now send data to Ernie securely, because data encrypted with the public key can only be decrypted with the private key (which only Ernie has). Bert uses this mechanism to send a shared secret. Bert and Ernie can now communicate with each other using a shared secret mechanism, confident in the knowledge that no third party has intercepted that secret. Alternately, Bert could give Ernie his public key, and they could both encrypt data using each other’s public keys, or more commonly by using those public keys to encrypt a session key and encrypting the data with that session key. Using Public Keys for Identity Verification Public key cryptography can also be used for verification of identity. Kyle wants to know if someone on the Internet who claims to be Stan is really Stan. A few months earlier, Stan handed Kyle his public key on a floppy disk. Thus, since Kyle already has Stan’s public key (and trusts the source of that key), he can now easily verify Stan’s identity. To achieve this, Kyle sends a cleartext message and asks Stan to encrypt it. Stan encrypts it with his private key. Kyle then uses Stan’s public key to decode the ciphertext. If the resulting cleartext matches, then the person on the other end must be Stan (unless someone else has Stan’s private key). Using Public Keys for Data Integrity Checking Finally, public key cryptography can be used for signing. Ahmed is in charge of meetings of a secret society called the Stupid Acronym Preventionists club. Abraham is a member of the club and gets a TIFF file containing a notice of their next meeting, passed on by way of a fellow member of the science club, Albert. Abraham is concerned, however, that the notice might have come from Bubba, who is trying to infiltrate the SAPs. Ahmed, however, was one step ahead, and took a checksum of the original message and encrypted the checksum with his private key, and sent the encrypted checksum as an attachment. Abraham used Ahmed’s public key to decrypt the checksum, and found that the checksum did not match that of the actual document. He wisely avoided the meeting. Isaac, however, was tricked into revealing himself as a SAP because he didn’t remember to check the signature on the message. Security Considerations Key-based Authentication and Encryption 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 35The moral of thisstory? One should always beware of geekssharing TIFFs—that is, if the security ofsome piece of data isimportant and if you do not have a direct,secure means of communication between two applications, computers, people, and so on, you must verify the authenticity of any communication using signatures, keys, or some other similar method. This may save your data and also save face. Encryption Summary Encryption is a powerful technique for keeping data secure if the initial key exchange occursin a secure fashion. One meansfor thisisto have a public key,stored in a well-known (and trusted) location. This allowsfor one-way encrypted communication through which a shared secret can be transferred for later two-way encrypted communication. You can use encryption not only for protecting data, but also for verifying the authenticity of data by encrypting a checksum. You can also use it to verify the identity of a client by requiring that the client encrypt some random piece of data as proof that the client holds the appropriate encryption key. Encryption, however, is not the final word in computer security. Because it depends on having some form of trusted key exchange, additional infrastructure is needed in order to achieve total security in environments where communication can be intercepted and modified. Console Debugging Warning: Failure to follow this advice can unintentionally expose security-critical information. In traditional UNIX and UNIX-like systems, the console is owned by root. Only root sees console messages. For this reason, print statements in the kernel are relatively secure. In OS X, any user can run the Console application. This represents a major departure from other UNIX-like systems. While it is never a good idea to include sensitive information in kernel debugging statements, it is particularly important not to do so in OS X. You must assume that any information displayed to the console could potentially be read by any user on the system (since the console is virtualized in the form of a user-viewable window). Printing any information involving sensitive data, including its location on disk or in memory, represents a security hole, however slight, and you should write your code accordingly. Obviously this is of less concern if that information is only printed when the user sets a debugging flag somewhere, but for normal use, printing potentially private information to the console is strongly discouraged. Security Considerations Console Debugging 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 36You must also be careful not to inadvertently print information that you use for generating password hashes or encryption keys, such as seed values passed to a random number generator. This is, by necessity, not a complete list of information to avoid printing to the console. You must use your own judgement when deciding whether a piece of information could be valuable if seen by a third party, and then decide if it is appropriate to print it to the console. Code Passing There are many ways of passing executable code into the kernel from user space. For the purposes of this section, executable code is not limited to compiled object code. It includes any instructions passed into the kernel that significantly affect control flow. Examples of passed-in executable code range from simple rules such as the filtering code uploaded in many firewall designs to bytecode uploads for a SCSI card. If it is possible to execute your code in user space, you should not even contemplate pushing code into the kernel. For the rare occasion where no other reasonable solution exists, however, you may need to pass some form of executable code into the kernel. This section explains some of the security ramifications of pushing code into the kernel and the level of verification needed to ensure consistent operation. Here are some guidelines to minimize the potential for security holes: 1. No raw object code. Direct execution of code passed in from user space is very dangerous. Interpreted languages are the only reasonable solution for this sort of problem, and even this is fraught with difficulty. Traditional machine code can’t be checked sufficiently to ensure security compliance. 2. Bounds checking. Since you are in the kernel, you are responsible for making sure that any uploaded code does not randomly access memory and does not attempt to do direct hardware access. You would normally make this a feature of the language itself, restricting access to the data element on which the bytecode is operating. 3. Termination checking. With very, very few exceptions, the language chosen should be limited to code that can be verified to terminate, and you should verify accordingly. If your driver is stuck in a tightly rolled loop, it is probably unable to do its job, and may impact overall system performance in the process. A language that does not allow (unbounded) loops (for example, allowing for but not while or goto could be one way to ensure termination. 4. Validity checking. Security Considerations Code Passing 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 37Your bytecode interpreter would be responsible for checking ahead for any potentially invalid operations and taking appropriate punitive actions against the uploaded code. For example, if uploaded code is allowed to do math, then proper protection must be in place to handle divide by zero errors. 5. Sanity checking. You should verify that the output is something remotely reasonable, if possible. It is not always possible to verify that the output is correct, but it is generally possible to create rules that prevent egregiously invalid output. For example, a network filter rule should output something resembling packets. If the checksums are bad, or if other information is missing or corrupt, clearly the uploaded code is faulty, and appropriate actions should be taken. It would be highly inappropriate for OS X to send out bad network traffic. In general, the more restrictive the language set, the lower the security risk. For example, interpreting simple network routing policies is less likely to be a security problem than interpreting packet rewriting rules, which is less likely to be an issue than running Java bytecode in the kernel. As with anything else, you must carefully weigh the potential benefits against the potential drawbacks and make the best decision given the information available. Security Considerations Code Passing 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 38Performance is a key aspect of any software system. Nowhere is this more true than in the kernel, where small performance problems tend to be magnified by repeated execution. For this reason, it is extremely important that your code be as efficient as possible. This chapter discusses the importance of low interrupt latency and fine-grained locking and tells you how to determine what portions of your code would benefit most from more efficient design. Interrupt Latency In OS X, you will probably never need to write code that runs in an interrupt context. In general, only motherboard hardware requires this. However, in the unlikely event that you do need to write code in an interrupt context, interrupt latency should be a primary concern. Interrupt latency refers to the delay between an interrupt being generated and an interrupt handler actually beginning to service that interrupt. In practice, the worst case interrupt latency is closely tied to the amount of time spent in supervisor mode (also called kernel mode) with interrupts off while handling some other interrupt. Low interrupt latency is necessary for reasonable overall performance, particularly when working with audio and video. In order to have reasonable soft real-time performance (for example, performance of multimedia applications), the interrupt latency caused by every device driver must be both small and bounded. OS X takes great care to bound and minimize interrupt latency for built-in drivers. It doesthis primarily through the use of interrupt service threads (also known as I/O service threads). When OS X takes an interrupt, the low-level trap handlers call up to a generic interrupt handling routine that clears the pending interrupt bit in the interrupt controller and calls a device-specific interrupt handler. That device-specific handler, in turn, sends a message to an interrupt service thread to notify it that an interrupt has occurred, and then the handler returns. When no further interrupts are pending, control returns to the currently executing thread. The next time the interrupt service thread is scheduled, it checks to see if an interrupt has occurred, then services the interrupt. As the name suggests, this actually is happening in a thread context, not an interrupt context. This design causes two major differences from traditional operating system design: ● Interrupt latency is near zero, since the code executing in an interrupt context is very small. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 39 Performance Considerations● It is possible for an interrupt to occur while a device driver is executing. This means that traditional (threaded) device drivers can be preempted and must use locking or other similar methods to protect any shared data (although they need to do so anyway to work on computers with multiple processors). This model is crucial to the performance of OS X. You should not attempt to circumvent this design by doing large amounts of work in an interrupt context. Doing so will be detrimental to the overall performance of the system. Locking Bottlenecks It is difficult to communicate data between multiple threads or between thread and interrupt contexts without using locking or other synchronization. This locking protects your data from getting clobbered by another thread. However, it also has the unfortunate side effect of being a potential bottleneck. In some types of communication (particularly n-way), locking can dramatically hinder performance by allowing only one thing to happen at a time. Read-write locks, discussed in “Synchronization Primitives” (page 128), can help alleviate this problem in the most common situation where multiple clients need to be able to read information but only rarely need to modify that data. However, there are many cases where read-write locks are not helpful. This section discusses some possible problems and ways of improving performance within those constraints. Working With Highly Contended Locks When many threads need to obtain a lock (or a small number of threads need to obtain a lock frequently), this lock is considered highly contended. Highly contended locks frequently represent faulty code design, but they are sometimes unavoidable. In those cases, the lock tends to become a major performance bottleneck. Take, for example, the issue of many-to-many communication that must be synchronized through a common buffer. While some improvement can be gained by using read-write locks instead of an ordinary mutex, the issue of multiple writers means that read-write locks still perform badly. One possible solution for this many-to-many communication problem is to break the lock up into multiple locks. Instead of sharing a single buffer for the communication itself, make a shared buffer that contains accounting information for the communication (for example, a list of buffers available for reading). Then assign each individual buffer its own lock. The readers might then need to check several locations to find the right data, but this still frequently yields better performance, since writers must only contend for a write lock while modifying the accounting information. Performance Considerations Locking Bottlenecks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 40Anothersolution for many-to-many communicationsisto eliminate the buffer entirely and communicate using message passing, sockets, IPC, RPC, or other methods. Yet another solution is to restructure your code in a way that the locking is unnecessary. This is often much more difficult. One method that is often helpful isto take advantage of flags and atomic increments, as outlined in the next paragraph. For simplicity, a single-writer, single-reader example is presented, but it is possible to extend this idea to more complicated designs. Take a buffer with some number of slots. Keep a read index and a write index into that buffer. When the write index and read index are the same, there is no data in the buffer. When writing, clear the next location. Then do an atomic increment on the pointer. Write the data. End by setting a flag at that new location that says that the data is valid. Note that this solution becomes much more difficult when dealing with multiple readers and multiple writers, and as such, is beyond the scope of this section. Reducing Contention by Decreasing Granularity One of the fundamental properties of locksis granularity. The granularity of a lock refersto the amount of code or data that it protects. A lock that protects a large block of code or a large amount of data is referred to as a coarse-grained lock, while a lock that protects only a small amount of code or data isreferred to as a fine-grained lock. A coarse-grained lock is much more likely to be contended (needed by one thread while being held by another) than a more finely grained lock. There are two basic ways of decreasing granularity. The first is to minimize the amount of code executed while a lock is held. For example, if you have code that calculates a value and stores it into a table, don’t take the lock before calling the function and release it after the function returns. Instead, take the lock in that piece of code right before you write the data, and release it as soon as you no longer need it. Of course, reducing the amount of protected code is not always possible or practical if the code needs to guarantee consistency where the value it is writing depends on other values in the table, since those values could change before you obtain the lock, requiring you to go back and redo the work. It is also possible to reduce granularity by locking the data in smaller units. In the above example, you could have a lock on each cell of the table. When updating cells in the table, you would start by determining the cells on which the destination cell depends, then lock those cells and the destination cell in some fixed order. (To avoid deadlock, you must always either lock cells in the same order or use an appropriate try function and release all locks on failure.) Once you have locked all the cells involved, you can then perform your calculation and release the locks, confident that no other thread has corrupted your calculations. However, by locking on a smaller unit of data, you have also reduced the likelihood of two threads needing to access the same cell. Performance Considerations Locking Bottlenecks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 41A slightly more radical version of this is to use read-write locks on a per-cell basis and always upgrade in a particular order. This is, however, rather extreme, and difficult to do correctly. Code Profiling Code profiling means determining how often certain pieces of code are executed. By knowing how frequently a piece of code is used, you can more accurately gauge the importance of optimizing that piece of code. There are a number of good tools for profiling user space applications. However, code profiling in the kernel is a very different beast, since it isn’t reasonable to attach to it like you would a running process. (It is possible by using a second computer, but even then, it is not a trivial task.) This section describes two useful ways of profiling your kernel code: counters and lock profiling. Any changes you make to allow code profiling should be done only during development. These are not the sort of changes that you want to release to end users. Using Counters for Code Profiling The first method of code profiling is with counters. To profile a section of code with a counter, you must first create a global variable whose name describesthat piece of code and initialize it to zero. You then add something like #ifdef PROFILING foo_counter++; #endif in the appropriate piece of code. If you then define PROFILING, that counter is created and initialized to zero, then incremented each time the code in question is executed. One small snag with this sort of profiling is the problem of obtaining the data. This can be done in several ways. The simplest is probably to install a sysctl, using the address of foo_counter as an argument. Then, you could simply issue the sysctl command from the command line and read or clear the variable. Adding a sysctl is described in more detail in “BSD sysctl API ” (page 117). In addition to using sysctl, you could also obtain the data by printing its value when unloading the module (in the case of a KEXT) or by using a remote debugger to attach to the kernel and directly inspecting the variable. However, a sysctl provides the most flexibility. With a sysctl, you can sample the value at any time, not just when the module is unloaded. The ability to arbitrarily sample the value makes it easier to determine the importance of a piece of code to one particular action. Performance Considerations Code Profiling 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 42If you are developing code for use in the I/O Kit, you should probably use your driver’s setProperties call instead of a sysctl. Lock Profiling Lock profiling is another useful way to find the cause of code inefficiency. Lock profiling can give you the following information: ● how many times a lock was taken ● how long the lock was held on average ● how often the lock was unavailable Put another way, this allows you to determine the contention of a lock, and in so doing, can help you to minimize contention by code restructuring. There are many different ways to do lock profiling. The most common way is to create your own lock calls that increment a counter and then call the real locking functions. When you move from debugging into a testing cycle before release, you can then replace the functions with defines to cause the actual functions to be called directly. For example, you might write something like this: extern struct timeval time; boolean_t mymutex_try(mymutex_t *lock) { int ret; ret=mutex_try(lock->mutex); if (ret) { lock->tryfailcount++; } return ret; } void mymutex_lock(mymutex_t *lock) { if (!(mymutex_try(lock))) { mutex_lock(lock->mutex); } lock->starttime = time.tv_sec; } void mymutex_unlock(mymutex_t *lock) { Performance Considerations Code Profiling 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 43lock->lockheldtime += (time.tv_sec - lock->starttime); lock->heldcount++; mutex_unlock(lock->mutex); } This routine has accuracy only to the nearest second, which is not particularly accurate. Ideally, you want to keep track of both time.tv_sec and time.tv_usec and roll the microseconds into seconds as the number gets large. From this information, you can obtain the average time the lock was held by dividing the total time held by the number of times it was held. It also tells you the number of times a lock was taken immediately instead of waiting, which is a valuable piece of data when analyzing contention. As with counter-based profiling, after you have written code to record lock use and contention, you must find a way to obtain that information. A sysctl is a good way of doing this, since it is relatively easy to implement and can provide a “snapshot” view of the data structure at any point in time. For more information on adding a sysctl, see “BSD sysctl API ” (page 117). Another way to do lock profiling isto use the built-in ETAP (Event Trace Analysis Package). This package consists of additional code designed for lock profiling. However, since this requires a kernel recompile, it is generally not recommended. Performance Considerations Code Profiling 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 44As described in “Keep Out” (page 13), programming in the kernel is fraught with hazards that can cause instability, crashes, or security holes. In addition to these issues, programming in the kernel has the potential for compatibility problems. If you program only to the interfaces discussed in this document or other Apple documents, you will avoid the majority of these. However, even limiting yourself to documented interfaces does not protect you from a handful of pitfalls. The biggest potential problem that you face is namespace collision, which occurs when your function, variable, or class name is the same as someone else’s. Since this makes one kernel extension or the other fail to load correctly (in a non-deterministic fashion), Apple has established function naming conventions for C and C++ code within the kernel. These are described in “Standard C Naming Conventions” (page 47) and “C++ Naming Conventions” (page 45), respectively. In addition to compatibility problems, kernel extensions that misbehave can also dramatically decrease the system’s overall performance or cause crashes. Some of these issues are described in “Performance and Stability Tips” (page 50). For more thorough coverage of performance and stability, you should also read the chapters “Security Considerations” (page 24) and “Performance Considerations” (page 39). C++ Naming Conventions Basic I/O Kit C++ naming conventions are defined in the document I/O Kit Device Driver Design Guidelines. This section refines those conventions in ways that should make them more useful to you as a programmer. Basic Conventions The primary conventions are as follows: ● Use the Java-style reverse DNS naming convention, substituting underscores for periods. For example, com_apple_foo. ● Avoid the following reserved prefixes: ● OS ● os ● IO ● io 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 45 Kernel Programming Style● Apple ● apple ● AAPL ● aapl This ensures that you will not collide with classes created by other companies or with future classes added to the operating system by Apple. It does not protect you from other projects created within your company, however, and for this reason, some additional guidelines are suggested. Additional Guidelines These additional guidelines are intended to minimize the chance of accidentally breaking your own software and to improve readability of code by developers. ● To avoid namespace collisions, you should prefix the names of classes and families with project names or other reasonably unique prefix codes. For example, if you are working on a video capture driver, and one of its classes is called capture, you will probably encounter a name collision eventually. Instead, you should name the class something like com_mycompany_driver_myproduct_capture. Similarly, names like To maximize readability, you should use macros to rename classes and families at compile time. For example: #define captureClass com_mycompany_driver_myproduct_capture #define captureFamily com_mycompany_iokit_myproduct_capture ● Use prefixes in function and method names to make it easier to see relationships between them. For example, Apple uses NS, CF, IO, and other prefixesto indicate that functions belong to specific frameworks. This might be as simple as prefixing a function with the name of the enclosing or related class, or it might be some other scheme that makes sense for your project. These are only suggested guidelines. Your company or organization should adopt its own set of guidelines within the constraints of the basic conventions described in the previous section. These guidelines should provide a good starting point. Kernel Programming Style C++ Naming Conventions 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 46Standard C Naming Conventions The naming conventionsfor C++ have been defined forsome time in the document I/O Kit Device Driver Design Guidelines. However, no conventions have been given for standard C code. Because standard C has an even greater chance of namespace collision than C++, it is essential that you follow these guidelines when writing C code for use in the kernel. Because C does not have the benefit of classes, it is much easier to run into a naming conflict between two functions. For this reason, the following conventions are suggested: ● Declare all functions and (global) variables static where possible to prevent them from being seen in the global namespace. If you need to share these across files within your KEXT, you can achieve a similar effect by declaring them __private_extern__. ● Each function name should use Java-style reverse DNS naming. For example, if your company is apple.com, you should begin each function with com_apple_. ● Follow the reverse DNS name with the name of your project. For example, if you work at Apple and were working on project Schlassen, you would start each function name (in drivers) with com_apple_driver_schlassen_. Note: The term driver is reserved for actual device drivers. For families, you should instead use iokit. For example, if project Schlassen is an I/O Kit family, function namesshould all begin with com_apple_iokit_schlassen_. ● Use hierarchical names if you anticipate multiple projects with similar names coming from different parts of your company or organization. ● Use macro expansion to save typing, for example PROJECT_eat could expand to com_apple_driver_schlassen_pickle_eat. ● If you anticipate that the last part of a function name may be the same as the last part of another function name (for example, PROJECT1_eat and PROJECT2_eat), you should change the namesto avoid confusion (for example, PROJECT1_eatpickle and PROJECT2_eatburger). ● Avoid the following reserved prefixes: ● OS ● os ● IO ● io ● Apple ● apple Kernel Programming Style Standard C Naming Conventions 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 47● AAPL ● aapl ● Avoid conflicting with any names already in the kernel, and do not use prefixes similar to those of existing kernel functions that you may be working with. ● Never begin a function name with an underscore (_). ● Under no circumstances should you use common names for your functions without prefixing them with the name of your project in some form. These are some examples of unacceptable names: ● getuseridentity ● get_user_info ● print ● find ● search ● sort ● quicksort ● merge ● console_log In short, picking any name that you would normally pick for a function is generally a bad idea, because every other developer writing code is likely to pick the same name for his or her function. Occasional conflicts are a fact of life. However, by following these few simple rules, you should be able to avoid the majority of common namespace pitfalls. Commonly Used Functions One of the most common problems faced when programming in the kernel is use of “standard” functions—things like printf or bcopy. Many commonly used standard C library functions are implemented in the kernel. In order to use them, however, you need to include the appropriate prototypes, which may be different from the user space prototypes for those functions, and which generally have different names when included from kernel code. In general, any non–I/O Kit header that you can safely include in the kernel is located in xnu/bsd/sys or xnu/osfmk/mach, although there are a few specialized headers in other places like libkern and libsa. Normal headers (those in /usr/include) cannot be used in the kernel. Kernel Programming Style Commonly Used Functions 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 48Important: If you are writing an I/O Kit KEXT, most of these functions are not what you are looking for. The I/O Kit providesits own APIsfor these features, including IOLog, IOMemoryDescriptor, and IOLock. While using the lower-level functionality is not expressly forbidden, it is generally discouraged (though printf is always fine). For more information about APIs available to I/O Kit KEXTs, see Kernel Framework Reference . Table 7-1 (page 49) lists some commonly used C functions, variables, and types, and gives the location of their prototypes. Table 7-1 Commonly used C functions Function name Header path printf Buffer cache functions (bread, bwrite, and brelse) Directory entries Error numbers Kernel special variables Spinlocks malloc Queues Random number generator bzero, bcopy, copyin, and copyout timeout and untimeout Various time functions Standard type declarations User credentials OS and system information If the standard C function you are trying to use is not in one of these files, chances are the function is not supported for use within the kernel, and you need to implement your code in another way. Kernel Programming Style Commonly Used Functions 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 49The symbols in these header files are divided among multiple symbol sets, depending on the technology area where they were designed to be used. To use these, you may have to declare dependencies on any of the following: ● com.apple.kernel—You should generally avoid this. ● com.apple.kernel.bsd—BSD portions of the kernel. ● com.apple.kernel.iokit—The I/O Kit. ● com.apple.kernel.libkern—General-purpose functions. ● com.apple.kernel.mach—Mach-specific APIs. ● com.apple.kpi.bsd—BSD portions of the kernel (v10.4 and later). ● com.apple.kernel.iokit—The I/O Kit (v10.4 and later). ● com.apple.kernel.libkern—General-purpose functions (v10.4 and later). ● com.apple.kernel.mach—Mach-specific APIs (v10.4 and later). ● com.apple.kpi.unsupported—Unsupported legacy functionality (v10.4 and later). Where possible, you should specify a dependency on the KPI version of these symbols. However, these symbols are only available in v10.4 and later. For the I/O Kit and libkern, this should make little difference. For other areas, such as network kernel extensions or file system KEXTs, you must use the KPI versions if you want your extension to load in OS X v10.4 and later. For a complete list of symbols in any of these dependencies, run nm on the binaries in /System/Library/Extensions/System.kext/PlugIns. Performance and Stability Tips This section includes some basic tips on performance and stability. You should read the sections on security and performance for additional information. These tips cover only style issues, not general performance or stability issues. Performance and Stability Tips Programming in the kernel is subject to a number of restrictions that do not exist in application programming. The first and most important is the stack size. The kernel has a limited amount of space allocated for thread stacks, which can cause problems if you aren’t aware of the limitation. This means the following: ● Recursion must be bounded (to no more than a few levels). ● Recursion should be rewritten as iterative routines where possible. Kernel Programming Style Performance and Stability Tips 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 50● Large stack variables(function local) are dangerous. Do not use them. This also appliesto large local arrays. ● Dynamically allocated variables are preferred (using malloc or equivalent) over local variables for objects more than a few bytes in size. ● Functions should have as few arguments as possible. ● Pass pointers to structures, not the broken out elements. ● Don’t use arguments to avoid using global or class variables. ● Do name global variables in a way that protects you from collision. ● C++ functions should be declared static. ● Functions not obeying these rules can cause a kernel panic, or in extreme cases, do not even compile. In addition to issues of stack size, you should also avoid doing anything that would generate unnecessary load such as polling a device or address. A good example is the use of mutexes rather than spinlocks. You should also structure your locks in such a way to minimize contention and to minimize hold times on the most highly contended locks. Also, since unused memory (and particularly wired memory) can cause performance degradation, you should be careful to deallocate memory when it is no longer in use, and you should never allocate large regions of wired memory. This may be unavoidable in some applications, but should be avoided whenever possible and disposed of at the earliest possible opportunity. Allocating large contiguous blocks of memory at boot time is almost never acceptable, because it cannot be released. There are a number of issues that you should consider when deciding whether to use floating point math or AltiVec vector math in the kernel. First, the kernel takes a speed penalty whenever floating-point math or AltiVec instructions are used in a system call context (or other similar mechanisms where a user thread executes in a kernel context), as floating-point and AltiVec registers are only maintained when they are in use. Note: In cases where altivec or floating point has already been used in user space in the calling thread, there is no additional penalty for using them in the kernel. Thus, for things like audio drivers, the above does not apply. In general, you should avoid doing using floating-point math or AltiVec instructions in the kernel unless doing so will result in a significant speedup. It is not forbidden, but is strongly discouraged. Second, AltiVec was not supported in the kernel prior to OS X v10.3. It was not possible to detect this support from within the kernel until a later 10.3 software update. If you must deploy your KEXT on earlier versions of OS X, you must either provide a non-AltiVec version of your code or perform the AltiVec instructions in user space. Kernel Programming Style Performance and Stability Tips 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 51Finally, AltiVec data stream instructions (dst, dstt, dstst, dss, and dssall) are not supported in the kernel, even for processors that support them in user space. Do not attempt to use them. If you decide to use AltiVec in the kernel, your code can determine whether the CPU supports AltiVec using the sysctlbyname call to get the hw.optional.altivec property. For more information, see “The sysctlbyname System Call” (page 123). Stability Tips ● Don’tsleep while holding resources(locks, for example). While thisis not forbidden, it isstrongly discouraged to avoid deadlock. ● Be careful to allocate and free memory with matching calls. For example, do not use allocation routines from the I/O Kit and deallocation routines from BSD. Likewise, do not use IOMallocContiguous with IOFreePageable. ● Use reference counts to avoid freeing memory that is still in use elsewhere. Be sure to deallocate memory when its reference count reaches zero, but not before. ● Lock objects before operating on them, even to change reference counts. ● Never dereference pointers without verifying that they are not NULL. In particular, never do this: int foo = *argptr; unless you have already verified that argptr cannot possibly be NULL. ● Test code in sections and try to think up likely edge cases for calculations. ● Never assume that your code will be run only on big endian processors. ● Never assume that the size of an instance of a type will never change. Always use sizeof if you need this information. ● Never assume that a pointer will always be the same size as an int or long. Style Summary Kernel programming style is very much a matter of personal preference, and it is not practical to programmatically enforce the guidelines in this chapter. However, we strongly encourage you to follow these guidelines to the maximum extent possible. These guidelines were created based on frequent problems reported by developers writing code in the kernel. No one can force you to use good style in your programming, but if you do not, you do so at your own peril. Kernel Programming Style Style Summary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 52The fundamental services and primitives of the OS X kernel are based on Mach 3.0. Apple has modified and extended Mach to better meet OS X functional and performance goals. Mach 3.0 was originally conceived as a simple, extensible, communications microkernel. It is capable of running as a stand–alone kernel, with other traditional operating-system servicessuch asI/O, file systems, and networking stacks running as user-mode servers. However, in OS X, Mach is linked with other kernel components into a single kernel address space. This is primarily for performance; it is much faster to make a direct call between linked components than it is to send messages or do remote procedure calls (RPC) between separate tasks. This modular structure results in a more robust and extensible system than a monolithic kernel would allow, without the performance penalty of a pure microkernel. Thusin OS X, Mach is not primarily a communication hub between clients and servers. Instead, its value consists of its abstractions, its extensibility, and its flexibility. In particular, Mach provides ● object-based APIs with communication channels (for example, ports) as object references ● highly parallel execution, including preemptively scheduled threads and support for SMP ● a flexible scheduling framework, with support for real-time usage ● a complete set of IPC primitives, including messaging, RPC, synchronization, and notification ● support for large virtual addressspaces,shared memory regions, and memory objects backed by persistent store ● proven extensibility and portability, for example across instruction set architectures and in distributed environments ● security and resource management as a fundamental principle of design; all resources are virtualized Mach Kernel Abstractions Mach provides a small set of abstractions that have been designed to be both simple and powerful. These are the main kernel abstractions: ● Tasks. The units of resource ownership; each task consists of a virtual addressspace, a portright namespace, and one or more threads. (Similar to a process.) 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 53 Mach Overview● Threads. The units of CPU execution within a task. ● Address space. In conjunction with memory managers, Mach implements the notion of a sparse virtual address space and shared memory. ● Memory objects. The internal units of memory management. Memory objectsinclude named entries and regions; they are representations of potentially persistent data that may be mapped into address spaces. ● Ports. Secure, simplex communication channels, accessible only via send and receive capabilities (known as port rights). ● IPC. Message queues, remote procedure calls, notifications, semaphores, and lock sets. ● Time. Clocks, timers, and waiting. At the trap level, the interface to most Mach abstractions consists of messages sent to and from kernel ports representing those objects. The trap-level interfaces (such as mach_msg_overwrite_trap) and message formats are themselves abstracted in normal usage by the Mach Interface Generator (MIG). MIG is used to compile procedural interfaces to the message-based APIs, based on descriptions of those APIs. Tasks and Threads OS X processes and POSIX threads (pthreads) are implemented on top of Mach tasks and threads, respectively. A thread is a point of control flow in a task. A task exists to provide resources for the threads it contains. This split is made to provide for parallelism and resource sharing. A thread ● is a point of control flow in a task. ● has access to all of the elements of the containing task. ● executes (potentially) in parallel with other threads, even threads within the same task. ● has minimal state information for low overhead. A task ● is a collection ofsystem resources. These resources, with the exception of the addressspace, are referenced by ports. These resources may be shared with other tasks if rights to the ports are so distributed. ● provides a large, potentially sparse address space, referenced by virtual address. Portions of this space may be shared through inheritance or external memory management. ● contains some number of threads. Mach Overview Tasks and Threads 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 54Note that a task has no life of its own—only threads execute instructions. When it is said that “task Y does X,” what is really meant is that “a thread contained within task Y does X.” A task is a fairly expensive entity. It exists to be a collection of resources. All of the threads in a task share everything. Two tasks share nothing without an explicit action (although the action is often simple) and some resources (such as port receive rights) cannot be shared between two tasks at all. A thread is a fairly lightweight entity. It is fairly cheap to create and has low overhead to operate. This is true because a thread has little state information (mostly its register state). Its owning task bears the burden of resource management. On a multiprocessor computer, it is possible for multiple threads in a task to execute in parallel. Even when parallelism is not the goal, multiple threads have an advantage in that each thread can use a synchronous programming style, instead of attempting asynchronous programming with a single thread attempting to provide multiple services. A thread is the basic computational entity. A thread belongs to one and only one task that defines its virtual address space. To affect the structure of the address space or to reference any resource other than the address space, the thread must execute a special trap instruction that causesthe kernel to perform operations on behalf of the thread or to send a message to some agent on behalf of the thread. In general, these traps manipulate resources associated with the task containing the thread. Requests can be made of the kernel to manipulate these entities: to create them, delete them, and affect their state. Mach provides a flexible framework for thread–scheduling policies. Early versions of OS X support both time-sharing and fixed-priority policies. A time-sharing thread’s priority is raised and lowered to balance its resource consumption against other time-sharing threads. Fixed-priority threads execute for a certain quantum of time, and then are put at the end of the queue of threads of equal priority. Setting a fixed priority thread’s quantum level to infinity allows the thread to run until it blocks, or until it is preempted by a thread of higher priority. High priority real-time threads are usually fixed priority. OS X also provides time constraint scheduling for real-time performance. This scheduling allows you to specify that your thread must get a certain time quantum within a certain period of time. Mach scheduling is described further in “Mach Scheduling and Thread Interfaces” (page 77). Ports, Port Rights, Port Sets, and Port Namespaces With the exception of the task’s virtual address space, all other Mach resources are accessed through a level of indirection known as a port. A port is an endpoint of a unidirectional communication channel between a client who requests a service and a server who providesthe service. If a reply isto be provided to such a service request, a second port must be used. This is comparable to a (unidirectional) pipe in UNIX parlance. Mach Overview Ports, Port Rights, Port Sets, and Port Namespaces 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 55In most cases, the resource that is accessed by the port (that is, named by it) is referred to as an object. Most objects named by a port have a single receiver and (potentially) multiple senders. That is, there is exactly one receive port, and at least one sending port, for a typical object such as a message queue. The service to be provided by an object is determined by the manager that receives the request sent to the object. It follows that the kernel is the receiver for ports associated with kernel-provided objects and that the receiver for ports associated with task-provided objects is the task providing those objects. For ports that name task-provided objects, it is possible to change the receiver of requests for that port to a different task, for example by passing the port to that task in a message. A single task may have multiple ports that refer to resources it supports. For that matter, any given entity can have multiple ports that represent it, each implying different sets of permissible operations. For example, many objects have a name port and a control port (sometimes called the privileged port). Access to the control port allows the object to be manipulated; access to the name port simply names the object so that you can obtain information about it or perform other non-privileged operations against it. Tasks have permissions to access ports in certain ways (send, receive, send-once); these are called port rights. A port can be accessed only via a right. Ports are often used to grant clients access to objects within Mach. Having the right to send to the object’sIPC port denotesthe right to manipulate the object in prescribed ways. As such, port right ownership is the fundamental security mechanism within Mach. Having a right to an object is to have a capability to access or manipulate that object. Port rights can be copied and moved between tasks via IPC. Doing so, in effect, passes capabilities to some object or server. One type of object referred to by a port is a port set. As the name suggests, a port set is a set of port rights that can be treated as a single unit when receiving a message or event from any of the members of the set. Port sets permit one thread to wait on a number of message and event sources, for example in work loops. Traditionally in Mach, the communication channel denoted by a port was always a queue of messages. However, OS X supports additional types of communication channels, and these new types of IPC object are also represented by ports and port rights. See the section “Interprocess Communication (IPC)” (page 58), for more details about messages and other IPC types. Ports and port rights do not have systemwide names that allow arbitrary ports or rights to be manipulated directly. Ports can be manipulated by a task only if the task has a port right in its port namespace. A port right is specified by a port name, an integer index into a 32-bit port namespace. Each task has associated with it a single port namespace. Tasks acquire port rights when another task explicitly insertsthem into its namespace, when they receive rights in messages, by creating objects that return a right to the object, and via Mach calls for certain special ports (mach_thread_self, mach_task_self, and mach_reply_port.) Mach Overview Ports, Port Rights, Port Sets, and Port Namespaces 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 56Memory Management As with most modern operating systems, Mach provides addressing to large, sparse, virtual address spaces. Runtime access is made via virtual addresses that may not correspond to locations in physical memory at the initial time of the attempted access. Mach is responsible for taking a requested virtual address and assigning it a corresponding location in physical memory. It does so through demand paging. A range of a virtual address space is populated with data when a memory object is mapped into that range. All data in an addressspace is ultimately provided through memory objects. Mach asksthe owner of a memory object (a pager) for the contents of a page when establishing it in physical memory and returns the possibly modified data to the pager before reclaiming the page. OS X includes two built-in pagers—the default pager and the vnode pager. The default pager handles nonpersistent memory, known as anonymous memory. Anonymous memory is zero-initialized, and it exists only during the life of a task. The vnode pager maps files into memory objects. Mach exports an interface to memory objects to allow their contents to be contributed by user-mode tasks. This interface is known as the External Memory Management Interface, or EMMI. The memory management subsystem exports virtual memory handles known as named entries or named memory entries. Like most kernel resources, these are denoted by ports. Having a named memory entry handle allows the owner to map the underlying virtual memory object or to pass the right to map the underlying object to others. Mapping a named entry in two different tasks results in a shared memory window between the two tasks, thus providing a flexible method for establishing shared memory. Beginning in OS X v10.1, the EMMI system was enhanced to support “portless” EMMI. In traditional EMMI, two Mach ports were created for each memory region, and likewise two ports for each cached vnode. Portless EMMI, in its initial implementation, replaces this with direct memory references (basically pointers). In a future release, ports will be used for communication with pagers outside the kernel, while using direct references for communication with pagers that reside in kernel space. The net result of these changes is that early versions of portless EMMI do not support pagers running outside of kernel space. This support is expected to be reinstated in a future release. Addressranges of virtual memory space may also be populated through direct allocation (using vm_allocate). The underlying virtual memory object is anonymous and backed by the default pager. Shared ranges of an address space may also be set up via inheritance. When new tasks are created, they are cloned from a parent. This cloning pertains to the underlying memory address space as well. Mapped portions of objects may be inherited as a copy, or asshared, or not at all, based on attributes associated with the mappings. Mach practices a form of delayed copy known as copy-on-write to optimize the performance of inherited copies on task creation. Mach Overview Memory Management 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 57Rather than directly copying the range, a copy-on-write optimization is accomplished by protected sharing. The two tasks share the memory to be copied, but with read-only access. When either task attempts to modify a portion of the range, that portion is copied at that time. Thislazy evaluation of memory copiesis an important optimization that permits simplifications in several areas, notably the messaging APIs. One other form of sharing is provided by Mach, through the export of named regions. A named region is a form of a named entry, but instead of being backed by a virtual memory object, it is backed by a virtual map fragment. This fragment may hold mappings to numerous virtual memory objects. It is mappable into other virtual maps, providing a way of inheriting not only a group of virtual memory objects but also their existing mapping relationships. This feature offers significant optimization in task setup, for example when sharing a complex region of the address space used for shared libraries. Interprocess Communication (IPC) Communication between tasksis an important element of the Mach philosophy. Mach supports a client/server system structure in which tasks(clients) accessservices by making requests of other tasks(servers) via messages sent over a communication channel. The endpoints of these communication channels in Mach are called ports, while port rights denote permission to use the channel. The forms of IPC provided by Mach include ● message queues ● semaphores ● notifications ● lock sets ● remote procedure calls (RPCs) The type of IPC object denoted by the port determines the operations permissible on that port, and how (and whether) data transfer occurs. Important: The IPC facilities in OS X are in a state of transition. In early versions of the system, not all of these IPC types may be implemented. There are two fundamentally different Mach APIs for raw manipulation of ports—the mach_ipc family and the mach_msg family. Within reason, both families may be used with any IPC object; however, the mach_ipc calls are preferred in new code. The mach_ipc calls maintain state information where appropriate in order to support the notion of a transaction. The mach_msg calls are supported for legacy code but deprecated; they are stateless. Mach Overview Interprocess Communication (IPC) 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 58IPC Transactions and Event Dispatching When a thread calls mach_ipc_dispatch, it repeatedly processes events coming in on the registered port set. These events could be an argument block from an RPC object (as the results of a client’s call), a lock object being taken (as a result of some other thread’s releasing the lock), a notification or semaphore being posted, or a message coming in from a traditional message queue. These events are handled via callouts from mach_msg_dispatch. Some events imply a transaction during the lifetime of the callout. In the case of a lock, the state is the ownership of the lock. When the callout returns, the lock is released. In the case of remote procedure calls, the state is the client’s identity, the argument block, and the reply port. When the callout returns, the reply is sent. When the callout returns, the transaction (if any) is completed, and the thread waits for the next event. The mach_ipc_dispatch facility is intended to support work loops. Message Queues Originally, the sole style of interprocess communication in Mach was the message queue. Only one task can hold the receive right for a port denoting a message queue. This one task is allowed to receive (read) messages from the port queue. Multiple tasks can hold rights to the port that allow them to send (write) messages into the queue. A task communicates with another task by building a data structure that contains a set of data elements and then performing a message-send operation on a port for which it holds send rights. At some later time, the task with receive rights to that port will perform a message-receive operation. A message may consist of some or all of the following: ● pure data ● copies of memory ranges ● port rights ● kernel implicit attributes, such as the sender’s security token The message transfer is an asynchronous operation. The message is logically copied into the receiving task, possibly with copy-on-write optimizations. Multiple threads within the receiving task can be attempting to receive messages from a given port, but only one thread can receive any given message. Semaphores Semaphore IPC objects support wait, post, and post all operations. These are counting semaphores, in that posts are saved (counted) if there are no threads currently waiting in that semaphore’s wait queue. A post all operation wakes up all currently waiting threads. Mach Overview Interprocess Communication (IPC) 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 59Notifications Like semaphores, notification objects also support post and wait operations, but with the addition of a state field. The state is a fixed-size, fixed-format field that is defined when the notification object is created. Each post updates the state field; there is a single state that is overwritten by each post. Locks A lock is an object that provides mutually exclusive access to a critical section. The primary interfaces to locks are transaction oriented (see “IPC Transactions and Event Dispatching” (page 59)). During the transaction, the thread holds the lock. When it returns from the transaction, the lock is released. Remote Procedure Call (RPC) Objects As the name implies, an RPC object is designed to facilitate and optimize remote procedure calls. The primary interfaces to RPC objects are transaction oriented (see “IPC Transactions and Event Dispatching” (page 59)) When an RPC object is created, a set of argument block formats is defined. When an RPC (a send on the object) is made by a client, it causes a message in one of the predefined formats to be created and queued on the object, then eventually passed to the server (the receiver). When the server returns from the transaction, the reply isreturned to the sender. Mach triesto optimize the transaction by executing the server using the client’s resources; this is called thread migration. Time Management The traditional abstraction of time in Mach is the clock, which provides a set of asynchronous alarm services based on mach_timespec_t. There are one or more clock objects, each defining a monotonically increasing time value expressed in nanoseconds. The real-time clock is built in, and is the most important, but there may be other clocksfor other notions of time in the system. Clockssupport operationsto get the current time,sleep for a given period, set an alarm (a notification that is sent at a given time), and so forth. The mach_timespec_t API is deprecated in OS X. The newer and preferred API is based on timer objects that in turn use AbsoluteTime as the basic data type. AbsoluteTime is a machine-dependent type, typically based on the platform-native time base. Routines are provided to convert AbsoluteTime values to and from other data types,such as nanoseconds. Timer objectssupport asynchronous, drift-free notification, cancellation, and premature alarms. They are more efficient and permit higher resolution than clocks. Mach Overview Time Management 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 60This chapter describes allocating memory and the low-level routinesfor modifying memory mapsin the kernel. It also describes a number of commonly used interfaces to the virtual memory system. It does not describe how to make changes in paging policy or add additional pagers. OS X does not support external pagers, although much of the functionality can be achieved in other ways, some of which are covered at a high level in this chapter. The implementation details of these interfaces are subject to change, however, and are thus left undocumented. With the exception of the section “Allocating Memory in the Kernel” (page 73), this chapter is of interest only if you are writing file systems or are modifying the virtual memory system itself. OS X VM Overview The VM system used in OS X is a descendent of Mach VM, which was created at Carnegie Mellon University in the 1980s. To a large extent, the fundamental design is the same, although some of the details are different, particularly when enhancing the VM system. It does, however, support the ability to request certain paging behavior through the use of universal page lists (UPLs). See “Universal Page Lists (UPLs)” (page 65) for more information. The design of Mach VM centers around the concept of physical memory being a cache for virtual memory. At its highest level, Mach VM consists of address spaces and ways to manipulate the contents of those address spaces from outside the space. These address spaces are sparse and have a notion of protections to limit what tasks can access their contents. At a lower level, the object level, virtual memory is seen as a collection of VM objects and memory objects, each with a particular owner and protections. These objects can be modified with object callsthat are available both to the task and (via the back end of the VM) to the pagers. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 61 Memory and Virtual MemoryNote: While memory objects and VM objects are closely related, the terms are not equivalent and should not be confused. .A VM object can be backed by one or more memory objects, which are, in turn, managed by a pager. A VM object may also be partially backed by other VM objects, as occurs in the case of shadow chains (described later in this section). The VM object is internal to the virtual memory system, and includes basic information about accessing the memory. The memory object, by contrast, is provided by the pager. The contents of the memory associated with that memory object can be retrieved from disk or some other backing store by exchanging messages with the memory object. Implicitly, each VM object is associated with a given pager through its memory object. VM objects are cached with system pages (RAM), which can be any power of two multiple of the hardware page size. In the OS X kernel,system pages are the same size as hardware pages. Each system page isrepresented in a given address space by a map entry. Each map entry has its own protection and inheritance. A given map entry can have an inheritance of shared, copy, or none. If a page is marked shared in a given map, child tasks share this page for reading and writing. If a page is marked copy, child tasks get a copy of this page (using copy-on-write). If a page is marked none, the child’s page is left unallocated. VM objects are managed by the machine-independent VM system, with the underlying virtual to physical mappings handled by the machine-dependent pmap system. The pmap system actually handles page tables, translation lookaside buffers, segments, and so on, depending on the design of the underlying hardware. When a VM object is duplicated (for example, the data pages from a process that has just called fork), a shadow object is created. A shadow object isinitially empty, and contains a reference to another object. When the contents of a page are modified, the page is copied from the parent object into the shadow object and then modified. When reading data from a page, if that page exists in the shadow object, the page listed in the shadow object is used. If the shadow object has no copy of that page, the original object is consulted. A series of shadow objects pointing to shadow objects or original objects is known as a shadow chain. Shadow chains can become arbitrarily long if an object is heavily reused in a copy-on-write fashion. However, since fork is frequently followed by exec, which replaces all of the material being shadowed, long chains are rare. Further, Mach automatically garbage collectsshadow objects, removing any intermediate shadow objects whose pages are no longer referenced by any (nondefunct) shadow object. It is even possible for the original object to be released if it no longer contains pages that are relevant to the chain. The VM calls available to an application include vm_map and vm_allocate, which can be used to map file data or anonymous memory into the address space. This is possible only because the address space is initially sparse. In general, an application can either map a file into its address space (through file mapping primitives, abstracted by BSD) or it can map an object (after being passed a handle to that object). In addition, a task can change the protections of the objects in its address space and can share those objects with other tasks. Memory and Virtual Memory OS X VM Overview 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 62In addition to the mapping and allocation aspects of virtual memory, the VM system contains a number of other subsystems. These include the back end (pagers) and the shared memory subsystem. There are also other subsystems closely tied to VM, including the VM shared memory server. These are described in “Other VM and VM-Related Subsystems” (page 68). Memory Maps Explained Each Mach task has its own memory map. In Mach, this memory map takes the form of an ordered doubly linked list. As described in “OS X VM Overview” (page 61), each of these objects contains a list of pages and shadow references to other objects. In general, you should never need to access a memory map directly unless you are modifying something deep within the VM system. The vm_map_entry structure contains task-specific information about an individual mapping along with a reference to the backing object. In essence, it is the glue between an VM object and a VM map. While the details of this data structure are beyond the scope of this document, a few fields are of particular importance. The field is_submap is a Boolean value that tells whether this map entry is a normal VM object or a submap. A submap is a collection of mappings that is part of a larger map. Submaps are often used to group mappings together for the purpose ofsharing them among multiple Mach tasks, but they may be used for many purposes. What makes a submap particularly powerful is that when several tasks have mapped a submap into their address space, they can see each other’s changes, not only to the contents of the objects in the map, but to the objects themselves. This means that as additional objects are added to or deleted from the submap, they appear in or disappear from the address spaces of all tasks that share that submap. The field behavior controls the paging reference behavior of a specified range in a given map. This value changes how pageins are clustered. Possible values are VM_BEHAVIOR_DEFAULT, VM_BEHAVIOR_RANDOM, VM_BEHAVIOR_SEQUENTIAL, and VM_BEHAVIOR_RSEQNTL, for default,random,sequential, orreverse-sequential pagein ordering. The protection and max_protection fields control the permissions on the object. The protection field indicates what rights the task currently has for the object, while the max_protection field contains the maximum access that the current task can obtain for the object. You might use the protection field when debugging shared memory. By setting the protection to be read-only, any inadvertent writes to the shared memory would cause an exception. However, when the task actually needsto write to thatshared region, it could increase its permissionsin the protection field to allow writes. Memory and Virtual Memory Memory Maps Explained 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 63It would be a security hole if a task could increase its own permissions on a memory object arbitrarily, however. In order to preserve a reasonable security model, the task that owns a memory object must be able to limit the rights granted to a subordinate task. For this reason, a task is not allowed to increase its protection beyond the permissions granted in max_protection. Possible valuesfor protection and max_protection are described in detail in xnu/osfmk/mach/vm_prot.h. Finally, the use_pmap field indicates whether a submap’s low-level mappings should be shared among all tasksinto which the submap is mapped. If the mappings are notshared, then the structure of the map isshared among all tasks, but the actual contents of the pages are not. For example,shared libraries are handled with two submaps. The read-only shared code section has use_pmap set to true. The read-write (nonshared) section has use_pmap set to false, forcing a clean copy of the library’s DATA segment to be mapped in from disk for each new task. Named Entries The OS X VM system provides an abstraction known as a named entry. A named entry is nothing more than a handle to a shared object or a submap. Shared memory support in OS X is achieved by sharing objects between the memory maps of various tasks. Shared memory objects must be created from existing VM objects by calling vm_allocate to allocate memory in your address space and then calling mach_make_memory_entry_64 to get a handle to the underlying VM object. The handle returned by mach_make_memory_entry_64 can be passed to vm_map to map that object into a given task’s address space. The handle can also be passed via IPC or other means to other tasks so that they can map it into their address spaces. This provides the ability to share objects with tasks that are not in your direct lineage, and also allows you to share additional memory with tasks in your direct lineage after those tasks are created. The other form of named entry, the submap, is used to group a set of mappings. The most common use of a submap is to share mappings among multiple Mach tasks. A submap can be created with vm_region_object_create. What makes a submap particularly powerful is that when several tasks have mapped a submap into their address space, they can see each other’s changes to both the data and the structure of the map. This means that one task can map or unmap a VM object in another task’s addressspace simply by mapping or unmapping that object in the submap. Memory and Virtual Memory Named Entries 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 64Universal Page Lists (UPLs) A universal page list, or UPL, is a data structure used when communicating with the virtual memory system. UPLs can be used to change the behavior of pages with respect to caching, permissions, mapping, and so on. UPLs can also be used to push data into and pull data from VM objects. The term is also often used to refer to the family of routines that operate on UPLs. The flags used when dealing with UPLs are described in osfmk/mach/memory_object_types.h. The life cycle of a UPL looks like this: 1. A UPL is created based on the contents of a VM object. This UPL includes information about the pages within that object. 2. That UPL is modified in some way. 3. The changes to the UPL are either committed (pushed back to the VM system) or aborted, with ubc_upl_commit or ubc_upl_abort, respectively. If you have a control handle for a given VM object (which generally means that you are inside a pager), you can use vm_object_upl_request to get a UPL for that object. Otherwise, you must use the vm_map_get_upl call. In either case, you are left with a handle to the UPL. When a pagein is requested, the pager receives a list of pages that are locked against the object, with certain pages set to not valid. The pager must either write data into those pages or must abort the transaction to prevent invalid data in the kernel. Similarly in pageout, the kernel must write the data to a backing store or abort the transaction to prevent data loss. The pager may also elect to bring additional pages into memory or throw additional pages out of memory at its discretion. Because pagers can be used both for virtual memory and for memory mapping of file data, when a pageout is requested, the data may need to be freed from memory, or it may be desirable to keep it there and simply flush the changes to disk. For this reason, the flag UPL_CLEAN_IN_PLACE exists to allow a page to be flushed to disk but not removed from memory. When a pager decides to page in or out additional pages, it must determine which pages to move. A pager can request all of the dirty pages by setting the RETURN_ONLY_DIRTY flag. It can also request all pages that are not in memory using the RETURN_ONLY_ABSENT flag. There is a slight problem, however. If a given page is marked as BUSY in the UPL, a request for information on that page would normally block. If the pager is doing prefetching or preflushing, this is not desirable, since it might be blocking on itself or on some other pager that is blocked waiting for the current transaction to complete. To avoid such deadlock, the UPL mechanism provides the UPL_NOBLOCK flag. This is frequently used in the anonymous pager for requesting free memory. Memory and Virtual Memory Universal Page Lists (UPLs) 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 65The flag QUERY_OBJECT_TYPE can be used to determine if an object is physically contiguous and to get other properties of the underlying object. The flag UPL_PRECIOUS means that there should be only one copy of the data. This prevents having a copy both in memory and in the backing store. However, this breaks the adjacency of adjacent pages in the backing store, and is thus generally not used to avoid a performance hit. The flag SET_INTERNAL is used by the BSD subsystem to cause all information about a UPL to be contained in a single memory object so that it can be passed around more easily. It can only be used if your code is running in the kernel’s address space. Since this handle can be used for multiple small transactions (for example, when mapping a file into memory block-by-block), the UPL API includes functions for committing and aborting changes to only a portion of the UPL. These functions are upl_commit_range and upl_abort_range, respectively. To aid in the use of UPLsfor handling multi-part transactions, the upl_commit_range and upl_abort_range calls have a flag that causes the UPL to be freed when there are no unmodified pages in the UPL. If you use this flag, you must be very careful not to use the UPL after all ranges have been committed or aborted. Finally, the function vm_map_get_upl is frequently used in file systems. It gets the underlying VM object associated with a given range within an address space. Since this returns only the first object in that range, it is your responsibility to determine whether the entire range is covered by the resulting UPL and, if not, to make additional calls to get UPLs for other objects. Note that while the vm_map_get_upl call is against an address space range, most UPL calls are against a vm_object. Using Mach Memory Maps Warning: Thissection describesthe low-level API for dealing with Mach VM maps. These maps cannot be modified in this way from a kernel extension. These functions are not available for use in a KEXT. They are presented strictly for use within the VM system and other parts of Mach. If you are not doing in-kernel development, you should be using the methods described in the chapter “Boundary Crossings” (page 109). From the context of the kernel (not from a KEXT), there are two maps that you will probably need to deal with. The first is the kernel map. Since your code is executing in the kernel’s address space, no additional effort is needed to use memory referenced in the kernel map. However, you may need to add additional mappings into the kernel map and remove them when they are no longer needed. Memory and Virtual Memory Using Mach Memory Maps 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 66The second map of interest is the memory map for a given task. This is of most interest for code that accepts input from user programs, for example a sysctl or a Mach RPC handler. In nearly all cases, convenient wrappers provide the needed functionality, however. Most of these functions are based around the vm_offset_t type, which is a pointer-sized integer. In effect, you can think of them as pointers, with the caveat that they are not necessarily pointers to data in the kernel’s address space, depending on usage. The low-level VM map API includes the following functions: kern_return_t vm_map_copyin(vm_map_t src_map, vm_offset_t src_addr, vm_size_t len, boolean_t src_destroy, vm_map_copy_t *copy_result); kern_return_t vm_map_copyout(vm_map_t map, vm_offset_t *addr, /* Out */ register vm_map_copy_t copy); kern_return_t vm_map_copy_overwrite(vm_map_t dst_map, vm_offset_t dst_address,vm_map_copy_t copy, boolean_t interruptible, pmap_t pmap); void vm_map_copy_discard(vm_map_copy_t copy); void vm_map_wire(vm_map_t map, vm_offset_t start, vm_offset_t end, vm_prot_t access_type, boolean_t user_wire); void vm_map_unwire(vm_map_t map, vm_offset_t start, vm_offset_t end, boolean_t user_wire); The function vm_map_copyin copies data from an arbitrary (potentially non–kernel) memory map into a copy list and returns the copy list pointer in copy_result. If something goes wrong and you need to throw away this intermediate object, it should be freed with vm_map_copy_discard. In order to actually get the data from the copy list, you need to overwrite a memory object in the kernel’s address space with vm_map_copy_overwrite. This overwrites an object with the contents of a copy list. For most purposes, the value passed for interruptible should be FALSE, and pmap should be NULL. Memory and Virtual Memory Using Mach Memory Maps 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 67Copying data from the kernel to user space is exactly the same as copying data from user space, except that you pass kernel_map to vm_map_copyin and pass the user map to vm_map_copy_overwrite. In general, however, you should avoid doing this, since you could end up with a task’s memory being fragmented into lots of tiny objects, which is undesirable. Do not use vm_map_copyout when copying data into an existing user task’s address map. The function vm_map_copyout is used for filling an unused region in an address map. If the region is allocated, then vm_map_copyout does nothing. Because it requires knowledge of the current state of the map, it is primarily used when creating a new address map (for example, if you are manually creating a new process). For most purposes, you do not need to use vm_map_copyout. The functions vm_map_wire and vm_map_unwire can be used to wire and unwire portions of an address map. If you set the argument user_wire to TRUE, then the page can be unwired from user space. This should be set to FALSE if you are about to use the memory for I/O or for some other operation that cannot tolerate paging. In vm_map_wire, the argument access_type indicates the types of accesses that should not be allowed to generate a page fault. In general, however, you should be using vm_wire to wire memory. As mentioned earlier, this information is presented strictly for use in the heart of the kernel. You cannot use anything in this section from a kernel extension. Other VM and VM-Related Subsystems There are two additional VM subsystems: pagers and the working set detection subsystem. In addition, the VM shared memory server subsystem is closely tied to (but is not part of) the VM subsystem. This section describes these three VM and VM-related subsystems. Pagers OS X has three basic pagers: the vnode pager, the default pager (or anonymous pager), and the device pager. These are used by the VM system to actually get data into the VM objects that underlie named entries. Pagers are linked into the VM system through a combination of a subset of the old Mach pager interface and UPLs. The default pager is what most people think of when they think of a VM system. It is responsible for moving normal data into and out of the backing store. In addition, there is a facility known as the dynamic pager that sits on top of the default pager and handles the creation and deletion of backing store files. These pager files are filled with data in clusters (groups of pages). When the total fullness of the paging file pool reaches a high–water mark, the default pager asks the dynamic pager to allocate a new store file. When the pool drops below its low water mark, the VM system selects a pager file, moves its contents into other pager files, and deletes it from disk. Memory and Virtual Memory Other VM and VM-Related Subsystems 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 68The vnode pager has a 1:1 (onto) mapping between objects in VM space and open files (vnodes). It is used for memory mapped file I/O. The vnode pager is generally hidden behind calls to BSD file APIs. The device pager allows you to map non–general-purpose memory with the cache characteristics required for that memory (WIMG). Non–general–purpose memory includes physical addresses that are mapped onto hardware other than main memory—for example, PCI memory, frame buffer memory, and so on. The device pager is generally hidden behind calls to various I/O Kit functions. Working Set Detection Subsystem To improve performance, OS X has a subsystem known asthe working set detection subsystem. Thissubsystem is called on a VM fault; it keeps a profile of the fault behavior of each task from the time of its inception. In addition, just before a page request, the fault code asksthissubsystem which adjacent pagesshould be brought in, and then makes a single large request to the pager. Since files on disk tend to have fairly good locality, and since address space locality is largely preserved in the backing store, this provides a substantial performance boost. Also, since it is based upon the application’s previous behavior, it tends to pull in pages that would probably have otherwise been needed later. This occurs for all pagers. The working set code works well once it is established. However, without help, its performance would be the baseline performance until a profile for a given application has been developed. To overcome this, the first time that an application is launched in a given user context, the initial working set required to start the application is captured and stored in a file. From then on, when the application is started, that file is used to seed the working set. These working set files are established on a per-user basis. They are stored in /var/vm/app_profile and are only accessible by the super-user (and the kernel). VM Shared Memory Server Subsystem The VM shared memory server subsystem is a BSD service that is closely tied to VM, but is not part of VM. This server provides two submaps that are used for shared library support in OS X. Because shared libraries contain both read-only portions (text segment) and read-write portions (data segment), the two portions are treated separately to maximize efficiency. The read-only portions are completely shared between tasks, including the underlying pmap entries. The read-write portions share a common submap, but have different underlying data objects (achieved through copy-on-write). The three functions exported by the VM shared memory server subsystem should only be called by dyld. Do not use them in your programs. Memory and Virtual Memory Other VM and VM-Related Subsystems 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 69The function load_shared_file is used to load a new shared library into the system. Once such a file is loaded, other tasks can then depend on it, so a shared library cannot be unshared. However, a new set of shared regions can be created with new_system_shared_regions so that no new tasks will use old libraries. The function reset_shared_file can be used to reset any changes that your task may have made to its private copy of the data section for a file. Finally, the function new_system_shared_regions can be used to create a new set of shared regions for future tasks. New regions can be used when updating prebinding with new shared libraries to cause new tasks to see the latest libraries at their new locations in memory. (Users of old shared libraries will still work, but they will fall off the pre-bound path and will perform less efficiently.) It can also be used when dealing with private libraries that you want to share only with your task’s descendents. Address Spaces This section explains issues that some developers may see when using their drivers in Panther or later. These changes were necessitated by a combination of hardware and underlying OS changes; however, you may see problems resulting from the changes even on existing hardware. There are three basic areas of change in OS X v10.3. These are: ● IOMemoryDescriptor changes ● VM system (pmap) changes ● Kernel dependency changes These are described in detail in the sections that follow. Background Info on PCI Address Translation To allow existing device drivers to work with upcoming 64-bit system architectures, a number of changes were required. To explain these, a brief introduction to PCI bus bridges is needed. When a PCI device needs to perform a data transaction to or from main memory, the device driver calls a series of functions intended to prepare this memory for I/O. In an architecture where both the device drivers and the memory subsystem use 32-bit addressing, everything just works, so long as the memory doesn't get paged out during the I/O operation. As kernel memory is generally not pageable, the preparation islargely superfluous. On a system whose memory subsystem uses 64-bit addressing, however, this becomes a bit of a problem. Because the hardware devices on the PCI bus can only handle 32-bit addresses, the device can only “see” a 4 gigabyte aperture into the (potentially much larger) main memory at any given time. Memory and Virtual Memory Address Spaces 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 70There are two possible solutionsfor this problem. The easy (butslow)solution would be to use “bounce buffers”. In such a design, device drivers would copy data into memory specifically allocated within the bottom 4 gigs of memory. However, this incurs a performance penalty and also puts additional constraints on the lower 4 gigs of memory, causing numerous problems for the VM system. The other solution, the one chosen in Apple's 64-bit implementation, is to use address translation to “map” blocks of memory into the 32-bit address space of the PCI devices. While the PCI device can still only see a 4 gig aperture, that aperture can then be non-contiguous, and thus bounce buffers and other restrictions are unnecessary. This address translation is done using a part of the memory controller known as DART, which stands for Device Address Resolution Table. This introduces a number of potential problems, however. First, physical addresses as seen by the processor no longer map 1:1 onto the addresses as seen by PCI devices. Thus, a new term, I/O addresses, is introduced to describe this new view. Because I/O addresses and physical addresses are no longer the same, the DART must keep a table of translations to use when mapping between them. Fortunately, if your driver is written according to Apple guidelines (using only documented APIs), this process is handled transparently. Note: This additional addressing mode has an impact when debugging I/O Kit device drivers. For more information, see “When Things Go Wrong: Debugging the Kernel” (page 161). IOMemoryDescriptor Changes When your driver calls IOMemoryDescriptor::prepare, a mapping is automatically injected into the DART. When it calls IOMemoryDescriptor::release , the mapping is removed. If you fail to do this, your driver could experience random data corruption or panics. Because the DART requires different caching for reads and writes, the DMA direction is important on hardware that includes a DART. While you may receive random failuresif the direction is wrong in general (on any system), if you attempt to call WriteBytes on a memory region whose DMA direction is set up for reading, you will cause a kernel panic on 64-bit hardware. If you attempt to perform a DMA transaction to unwired (user) memory, on previous systems, you would only get random crashes, panics, and data corruption. On machines with a DART, you will likely get no data whatsoever. As a side-effect of changes in the memory subsystem, OS X is much more likely to return physically contiguous page ranges in memory regions. Historically, OS X returned multi-page memory regions in reverse order, starting with the last page and moving towards the first page. The result of this was that multi-page memory regions essentially never had a contiguous range of physical pages. Memory and Virtual Memory Address Spaces 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 71Because of the increased probability of seeing physically contiguous blocks of memory in a memory region, this change may expose latent bugs in some drivers that only show up when handling contiguous ranges of physical pages, which could result in incorrect behavior or panics. Note that the problems mentioned above are caused by bugs in the drivers, and could result in problems on older hardware prior to Panther. These issues are more likely to occur in Panther and later versions of OS X, however, because of the new hardware designs and the OS changes that were made to support those designs. VM System and pmap Changes: In Panther, as a result of the changes described in detail in the section on PCI address translation, physical addresses obtained directly from the pmap layer have no useful purpose outside the VM system itself. To prevent their inadvertent use in device drivers, the pmap calls are no longer available from kernel extensions. A few drivers written prior to the addition of the IOMemoryDescriptor class still use pmap calls to get the physical pages associated with a virtual address. Also, a few developers have looked at the IOMemoryDescriptor implementation and chosen to obtain addresses directly from the pmap layer to remove what was perceived as an unnecessary abstraction layer. Even without removing access to the pmap calls, these drivers would not function on systems with a DART (see the PCI section above for info on DARTs). To better emphasize this upcoming failure, Panther will cause these drivers to fail to load with an undefined symbol error (generally for pmap_extract ) even on systems without a DART. Kernel Dependency Changes Beginning in Panther, device drivers that declare a dependency on version 7 (the Panther version) of the I/O Kit will no longer automatically get symbols from Mach and BSD. This change was made to discourage I/O Kit developers from relying on symbols that are not explicitly approved for use in the I/O Kit. Existing drivers are unaffected by this change. This change only affects you if you explicitly modify your device driver to declare a dependency on version 7 of the I/O Kit to take advantage of new I/O Kit features. Summary As described above, some device drivers may require minor modifications to support Panther and higher. Apple has made every effort to ensure compatibility with existing device driversto the greatest extent possible, but a few drivers may break. If your driver breaks, you should first check to see if your driver includes any of the bugs described in the previous sections. If it does not, contact Apple Developer Technical Support for additional debugging suggestions. Memory and Virtual Memory Address Spaces 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 72Allocating Memory in the Kernel As with most things in the OS X kernel, there are a number of ways to allocate memory. The choice of routines depends both on the location of the calling routine and on the reason for allocating memory. In general, you should use Mach routines for allocating memory unless you are writing code for use in the I/O Kit, in which case you should use I/O Kit routines. Allocating Memory From a Non-I/O-Kit Kernel Extension The header defines the following routines for kernel memory allocation: ● OSMalloc—allocates a block of memory. ● OSMalloc_noblock—allocates a block of memory, but immediately returns NULL if the request would block. ● OSMalloc_nowait—same as OSMalloc_noblock. ● OSFree—releases memory allocated with any of the OSMalloc variants. ● OSMalloc_Tagalloc—allows you to create a unique tag for your memory allocations. You must create at least one tag before you can use any of the OSMalloc functions. ● OSMalloc_Tagfree—releases a tag allocated with OSMalloc_Tagalloc. (You must release all allocations associated with that tag before you call this function.) For example, to allocate and free a page of wired memory, you might write code like this: #include #define MYTAGNAME "com.apple.mytag" ... OSMallocTag mytag = OSMalloc_Tagalloc(MYTAGNAME, OSMT_DEFAULT); void *datablock = OSMalloc(PAGE_SIZE_64, mytag); ... OSFree(datablock, PAGE_SIZE_64, mytag); To allocate a page of pageable memory, pass OSMT_PAGEABLE instead of OSMT_DEFAULT in your call to OSMalloc_Tagalloc. Memory and Virtual Memory Allocating Memory in the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 73Allocating Memory From the I/O Kit Although the I/O Kit is generally beyond the scope of this document, the I/O Kit memory management routines are presented here for completeness. In general, I/O Kit routinesshould not be used outside the I/O Kit. Similarly, Mach allocation routines should not be directly used from the I/O Kit because the I/O Kit has abstractions for those routines that fit the I/O Kit development model more closely. The I/O Kit includes the following routines for kernel memory allocation: void *IOMalloc(vm_size_t size); void *IOMallocAligned(vm_size_t size, vm_size_t alignment); void *IOMallocContiguous(vm_size_t size, vm_size_t alignment, IOPhysicalAddress *physicalAddress); void *IOMallocPageable(vm_size_t size, vm_size_t alignment); void IOFree(void *address, vm_size_t size); void IOFreeAligned(void *address, vm_size_t size); void IOFreeContiguous(void *address, vm_size_t size); void IOFreePageable(void *address, vm_size_t size); Most of these routines are relatively transparent wrappers around the Mach allocation functions. There are two major differences, however. First, the caller does not need to know which memory map is being modified. Second, they have a separate free call for each allocation call for internal bookkeeping reasons. The functions IOMallocContiguous and IOMallocAligned differsomewhat fromtheir Mach underpinnings. IOMallocAligned uses calls directly to Mach VM to add support for arbitrary (power of 2) data alignment, rather than aligning based on the size of the object. IOMallocContiguous adds an additional parameter, PhysicalAddress. If this pointer is not NULL, the physical address is returned through this pointer. Using Mach functions, obtaining the physical address requires a separate function call. Memory and Virtual Memory Allocating Memory in the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 74Important: If your KEXT allocates memory that will be shared, you should create a buffer of type IOMemoryDescriptor or IOBufferMemoryDescriptor and specify that the buffer should be sharable. If you are allocating memory in a user application that will be shared with the kernel, you should use valloc or vm_allocate instead of malloc and then call mach_make_memory_entry_64. Allocating Memory In the Kernel Itself In addition to the routines available to kernel extensions, there are a number of other functions you can call to allocate memory when you are modifying the Mach kernel itself. Mach routines provide a relatively straightforward interface for allocating and releasing memory. They are the preferred mechanism for allocating memory outside of the I/O Kit. BSD also offers _MALLOC and _FREE, which may be used in BSD parts of the kernel. These routines do not provide for forced mapping of a given physical address to a virtual address. However, if you need such a mapping, you are probably writing a device driver, in which case you should be using I/O Kit routines instead of Mach routines. Most of these functions are based around the vm_offset_t type, which is a pointer-sized integer. In effect, you can think of them as pointers, with the caveat that they are not necessarily pointers to data in the kernel’s address space, depending on usage. These are some of the commonly used Mach routines for allocating memory: kern_return_t kmem_alloc(vm_map_t map, vm_offset_t *addrp, vm_size_t size); void kmem_free(vm_map_t map, vm_offset_t addr, vm_size_t size); kern_return_t mem_alloc_aligned(vm_map_t map, vm_offset_t *addrp, vm_size_t size); kern_return_t kmem_alloc_wired(vm_map_t map, vm_offset_t *addrp, vm_size_t size); kern_return_t kmem_alloc_pageable(vm_map_t map, vm_offset_t *addrp, vm_size_t size); kern_return_t kmem_alloc_contig(vm_map_t map, vm_offset_t *addrp, vm_size_t size, vm_offset_t mask, int flags); These functions all take a map as the first argument. Unless you need to allocate memory in a different map, you should pass kernel_map for this argument. Memory and Virtual Memory Allocating Memory in the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 75All of the kmem_alloc functions except kmem_alloc_pageable allocate wired memory. The function kmem_alloc_pageable creates the appropriate VM structures but does not back the region with physical memory. This function could be combined with vm_map_copyout when creating a new address map, for example. In practice, it is rarely used. The function kmem_alloc_aligned allocates memory aligned according to the value of the size argument, which must be a power of 2. The function kmem_alloc_wired is synonymous with kmem_alloc and is appropriate for data structures that cannot be paged out. It is not strictly necessary; however, if you explicitly need certain pieces of data to be wired, using kmem_alloc_wired makes it easier to find those portions of your code. The function kmem_alloc_contig attempts to allocate a block of physically contiguous memory. This is not always possible, and requires a full sort of the system free list even for short allocations. After startup, this sort can cause long delays, particularly on systems with lots of RAM. You should generally not use this function. The function kmem_free is used to free an object allocated with one of the kmem_alloc functions. Unlike the standard C free function, kmem_free requires the length of the object. If you are not allocating fixed-size objects (for example, sizeof struct foo), you may have to do some additional bookkeeping, since you must free an entire object, not just a portion of one. Memory and Virtual Memory Allocating Memory in the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 76OS X is based on Mach and BSD. Like Mach and most BSD UNIX systems, it contains an advanced scheduler based on the CMU Mach 3 scheduler. This chapter describes the scheduler from the perspective of both a kernel programmer and an application developer attempting to set scheduling parameters. This chapter begins with the “Overview of Scheduling” (page 77), which describes the basic concepts behind Mach scheduling at a high level, including real-time priority support. The second section, “Using Mach Scheduling From User Applications” (page 79), describes how to access certain key Mach scheduler routines from user applications and from other parts of the kernel outside the scheduler. The third section, “Kernel Thread APIs” (page 85), explains scheduler-related topics including how to create and terminate kernel threads and describes the BSD spl macros and their limited usefulness in OS X. Overview of Scheduling The OS X scheduler is derived from the scheduler used in OSFMK 7.3. In general, much documentation about prior implementations applies to the scheduler in OS X, although you will find numerous differences. The details of those differences are beyond the scope of this overview. Mach scheduling is based on a system of run queues at various priorities that are handled in different ways. The priority levels are divided into four bands according to their characteristics, as described in Table 10-1 (page 77). Table 10-1 Thread priority bands Priority Band Characteristics Normal normal application thread priorities System high priority threads whose priority has been raised above normal threads reserved for threads created inside the kernel that need to run at a higher priority than all user space threads (I/O Kit workloops, for example) Kernel mode only threads whose priority is based on getting a well-defined fraction of total clock cycles, regardless of other activity (in an audio player application, for example). Real-time threads 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 77 Mach Scheduling and Thread InterfacesThreads can migrate between priority levels for a number of reasons, largely as an artifact of the time sharing algorithm used. However, this migration is within a given band. Threads marked as being real-time priority are also special in the eyes of the scheduler. A real-time thread tells the scheduler that it needs to run for A cycles out of the next B cycles. For example, it might need to run for 3000 out of the next 7000 clock cyclesin order to keep up. It also tellsthe scheduler whether those cycles must be contiguous. Using long contiguous quanta is generally frowned upon but is occasionally necessary for specialized real-time applications. The kernel will make every effort to honor the request, but since this is soft real-time, it cannot be guaranteed. In particular, if the real-time thread requests something relatively reasonable, its priority will remain in the real-time band, but if it lies blatantly about its requirements and behaves in a compute-bound fashion, it may be demoted to the priority of a normal thread. Changing a thread’s priority to turn it into a real-time priority thread using Mach calls is described in more detail in “Using Mach Scheduling From User Applications” (page 79). In addition to the raw Mach RPC interfaces, some aspects of a thread’s priority can be controlled from user space using the POSIX thread priority API. The POSIX thread API is able to set thread priority only within the lowest priority band (0–63). For more information on the POSIX thread priority API, see “Using the pthreads API to Influence Scheduling” (page 79). Why Did My Thread Priority Change? There are many reasons that a thread’s priority can change. This section attempts to explain the root cause of these thread priority changes. A real-time thread, as mentioned previously, is penalized (and may even be knocked down to normal thread priority) if it exceeds its time quantum without blocking repeatedly. For this reason, it is very important to make a reasonable guess about your thread’s workload if it needs to run in the real-time band. Threadsthat are heavily compute-bound are given lower priority to help minimize response time for interactive tasksso that high–priority compute–bound threads cannot monopolize the system and prevent lower–priority I/O-bound threads from running. Even at a lower priority, the compute–bound threads still run frequently, since the higher–priority I/O-bound threads do only a short amount of processing, block on I/O again, then allow the compute-bound threads to execute. All of these mechanisms are operating continually in the Mach scheduler. This meansthat threads are frequently moving up or down in priority based upon their behavior and the behavior of other threads in the system. Mach Scheduling and Thread Interfaces Why Did My Thread Priority Change? 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 78Using Mach Scheduling From User Applications There are three basic ways to change how a user thread is scheduled. You can use the BSD pthreads API to change basic policy and importance. You can also use Mach RPC calls to change a task’s importance. Finally, you can use RPC calls to change the scheduling policy to move a thread into a different scheduling band. This is commonly used when interacting with CoreAudio. The pthreads API is a user space API, and has limited relevance for kernel programmers. The Mach thread and task APIs are more general and can be used from anywhere in the kernel. The Mach thread and task calls can also be called from user applications. Using the pthreads API to Influence Scheduling OS X supports a number of policies at the POSIX threads API level. If you need real-time behavior, you must use the Mach thread_policy_set call. This is described in “Using the Mach Thread API to Influence Scheduling” (page 80). The pthreads API adjuststhe priority of threads within a given task. It does not necessarily impact performance relative to threads in other tasks. To increase the priority of a task, you can use nice or renice from the command line or call getpriority and setpriority from your application. The API providestwo functions: pthread_getschedparam and pthread_setschedparam. Their prototypes look like this: pthread_setschedparam(pthread_t thread, int policy, struct sched_param *param); pthread_getschedparam(pthread_t thread, int *policy, struct sched_param *param) The arguments for pthread_getschedparam are straightforward. The first argument is a thread ID, and the others are pointers to memory where the results will be stored. The argumentsto pthread_setschedparam are not as obvious, however. As with pthread_getschedparam, the first argument is a thread ID. The second argument to pthread_setschedparam is the desired policy, which can currently be one of SCHED_FIFO (first in, first out), SCHED_RR (round-robin), or SCHED_OTHER. The SCHED_OTHER policy is generally used for extra policies that are specific to a given operating system, and should thus be avoided when writing portable code. The third argument is a structure that contains various scheduling parameters. Mach Scheduling and Thread Interfaces Using Mach Scheduling From User Applications 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 79Here is a basic example of using pthreads functions to set a thread’s scheduling policy and priority. int set_my_thread_priority(int priority) { struct sched_param sp; memset(&sp, 0, sizeof(struct sched_param)); sp.sched_priority=priority; if (pthread_setschedparam(pthread_self(), SCHED_RR, &sp) == -1) { printf("Failed to change priority.\n"); return -1; } return 0; } This code snippet sets the scheduling policy for the current thread to round-robin scheduling, and sets the thread’s relative importance within the task to the value passed in through the priority argument. For more information, see the manual page for pthread. Using the Mach Thread API to Influence Scheduling This API is frequently used in multimedia applications to obtain real-time priority. It is also useful in other situations when the pthread scheduling API cannot be used or does not provide the needed functionality. The API consists of two functions, thread_policy_set and thread_policy_get. kern_return_t thread_policy_set( thread_act_t thread, thread_policy_flavor_t flavor, thread_policy_t policy_info, mach_msg_type_number_t count); kern_return_t thread_policy_get( thread_act_t thread, thread_policy_flavor_t flavor, thread_policy_t policy_info, mach_msg_type_number_t *count, Mach Scheduling and Thread Interfaces Using Mach Scheduling From User Applications 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 80boolean_t *get_default); The parameters of these functions are roughly the same, except that the thread_policy_get function takes pointers for the count and the get_default arguments. The count is an inout parameter, meaning that it is interpreted as the maximum amount of storage (in units of int32_t) that the calling task has allocated for the return, but it is also overwritten by the scheduler to indicate the amount of data that was actually returned. These functions get and set several parameters, according to the thread policy chosen. The possible thread policies are listed in Table 10-2 (page 81). Table 10-2 Thread policies Policy Meaning THREAD_STANDARD_POLICY Default value THREAD_TIME_CONSTRAINT_POLICY Used to specify real-time behavior. Used to indicate the importance of computation relative to other threads in a given task. THREAD_PRECEDENCE_POLICY The following code snippet shows how to set the priority of a task to tell the scheduler that it needs real-time performance. The example values provided in comments are based on the estimated needs of esd (the Esound daemon). #include #include #include #include int set_realtime(int period, int computation, int constraint) { struct thread_time_constraint_policy ttcpolicy; int ret; thread_port_t threadport = pthread_mach_thread_np(pthread_self()); ttcpolicy.period=period; // HZ/160 ttcpolicy.computation=computation; // HZ/3300; ttcpolicy.constraint=constraint; // HZ/2200; ttcpolicy.preemptible=1; Mach Scheduling and Thread Interfaces Using Mach Scheduling From User Applications 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 81if ((ret=thread_policy_set(threadport, THREAD_TIME_CONSTRAINT_POLICY, (thread_policy_t)&ttcpolicy, THREAD_TIME_CONSTRAINT_POLICY_COUNT)) != KERN_SUCCESS) { fprintf(stderr, "set_realtime() failed.\n"); return 0; } return 1; } The time values are in terms of Mach absolute time units. Since these values differ on different CPUs, you should generally use numbers relative to HZ (a global variable in the kernel that contains the current number of ticks per second). You can either handle this conversion yourself by dividing this value by an appropriate quantity or use the conversion routines described in “Using Kernel Time Abstractions ” (page 142). Say your computer reports 133 million for the value of HZ. If you pass the example values given as arguments to this function, your thread tells the scheduler that it needs approximately 40,000 (HZ/3300) out of the next 833,333 (HZ/160) bus cycles. The preemptible value (1) indicates that those 40,000 bus cycles need not be contiguous. However, the constraint value (HZ/2200) tells the scheduler that there can be no more than 60,000 bus cycles between the start of computation and the end of computation. Note: Because the constraint sets a maximum bound for computation, it must be larger than the value for computation. A straightforward example using this API is code that displays video directly to the framebuffer hardware. It needs to run for a certain number of cycles every frame to get the new data into the frame buffer. It can be interrupted without worry, but if it isinterrupted for too long, the video hardware starts displaying an outdated frame before the software writes the updated data, resulting in a nasty glitch. Audio has similar behavior, but since it is usually buffered along the way (in hardware and in software), there is greater tolerance for variations in timing, to a point. Another policy call is THREAD_PRECEDENCE_POLICY. This is used for setting the relative importance of non-real-time threads. Its calling convention issimilar, except that itsstructure is thread_precedence_policy, and contains only one field, an integer_t called importance. While thisis a signed 32-bit value, the minimum legal value is zero (IDLE_PRI). threads set to IDLE_PRI will only execute when no other thread is scheduled to execute. Mach Scheduling and Thread Interfaces Using Mach Scheduling From User Applications 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 82In general, larger values indicate higher priority. The maximum limit is subject to change, as are the priority bands, some of which have special purposes (such as real-time threads). Thus, in general, you should use pthreads APIs to achieve this functionality rather than using this policy directly unless you are setting up an idle thread. Using the Mach Task API to Influence Scheduling This relatively simple API is not particularly useful for most developers. However, it may be beneficial if you are developing a graphical user interface for Darwin. It also provides some insight into the prioritization of tasks in OS X. It is presented here for completeness. The API consists of two functions, task_policy_set and task_policy_get. kern_return_t task_policy_set( task_t task, task_policy_flavor_t flavor, task_policy_t policy_info, mach_msg_type_number_t count); kern_return_t task_policy_get( task_t task, task_policy_flavor_t flavor, task_policy_t policy_info, mach_msg_type_number_t *count, boolean_t *get_default); As with thread_policy_set and thread_policy_get, the parameters are similar, except that the task_policy_get function takes pointers for the count and the get_default arguments. The count argument is an inout parameter. It is interpreted as the maximum amount of storage that the calling task has allocated for the return, but it is also overwritten by the scheduler to indicate the amount of data that was actually returned. These functions get and set a single parameter, that of the role of a given task, which changes the way the task’s priority gets altered over time. The possible roles of a task are listed in Table 10-3 (page 83). Table 10-3 Task roles Role Meaning TASK_UNSPECIFIED Default value Mach Scheduling and Thread Interfaces Using Mach Scheduling From User Applications 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 83Role Meaning This is set when a process is executed with nice or is modified by renice. TASK_RENICED GUI application in the foreground. There can be more than one foreground application. TASK_FOREGROUND_APPLICATION TASK_BACKGROUND_APPLICATION GUI application in the background. TASK_CONTROL_APPLICATION Reserved for the dock or equivalent (assigned FCFS). TASK_GRAPHICS_SERVER Reserved for WindowServer or equivalent (assigned FCFS). The following code snippet shows how to set the priority of a task to tell the scheduler that it is a foreground application (regardless of whether it really is). #include #include #include int set_my_task_policy(void) { int ret; struct task_category_policy tcatpolicy; tcatpolicy.role = TASK_FOREGROUND_APPLICATION; if ((ret=task_policy_set(mach_task_self(), TASK_CATEGORY_POLICY, (thread_policy_t)&tcatpolicy, TASK_CATEGORY_POLICY_COUNT)) != KERN_SUCCESS) { fprintf(stderr, "set_my_task_policy() failed.\n"); return 0; } return 1; } Mach Scheduling and Thread Interfaces Using Mach Scheduling From User Applications 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 84Kernel Thread APIs The OS X scheduler provides a number of public APIs. While many of these APIs should not be used, the APIs to create, destroy, and alter kernel threads are of particular importance. While not technically part of the scheduler itself, they are inextricably tied to it. The scheduler directly provides certain services that are commonly associated with the use of kernel threads, without which kernel threads would be of limited utility. For example, the scheduler provides support for wait queues, which are used in various synchronization primitives such as mutex locks and semaphores. Creating and Destroying Kernel Threads The recommended interface for creating threads within the kernel is through the I/O Kit. It provides IOCreateThread, IOThreadSelf, and IOExitThread functions that make it relatively painless to create threads in the kernel. The basic functions for creating and terminating kernel threads are: IOThread IOCreateThread(IOThreadFunc function, void *argument); IOThread IOThreadSelf(void); void IOExitThread(void); With the exception of IOCreateThread (which is a bit more complex), the I/O Kit functions are fairly thin wrappers around Mach thread functions. The types involved are also very thin abstractions. IOThread is really the same as thread_t. The IOCreateThread function creates a new thread that immediately begins executing the function that you specify. It passes a single argument to that function. If you need to pass more than one argument, you should dynamically allocate a data structure and pass a pointer to that structure. For example, the following code creates a kernel thread and executes the function myfunc in that thread: #include #include #include struct mydata { int three; char *string; }; Mach Scheduling and Thread Interfaces Kernel Thread APIs 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 85static void myfunc(void *myarg) { struct mydata *md = (struct mydata *)myarg; IOLog("Passed %d = %s\n", md->three, md->string); IOExitThread(); } void start_threads() { IOThread mythread; struct mydata *md = (struct mydata *)malloc(sizeof(*md)); md->three = 3; md->string = (char *)malloc(2 * sizeof(char)); md->string[0] = '3'; md->string[1] = '\0'; // Start a thread using IOCreateThread mythread = IOCreateThread(&myfunc, (void *)md); } One other useful function is thread_terminate. This can be used to destroy an arbitrary thread (except, of course, the currently running thread). This can be extremely dangerous if not done correctly. Before tearing down a thread with thread_terminate, you should lock the thread and disable any outstanding timers against it. If you fail to deactivate a timer, a kernel panic will occur when the timer expires. With that in mind, you may be able to terminate a thread as follows: thread_terminate(getact_thread(thread)); There thread is of type thread_t. In general, you can only be assured that you can kill yourself, not other threads in the system. The function thread_terminate takes a single parameter of type thread_act_t (a thread activation). The function getact_thread takes a thread shuttle (thread_shuttle_t) or thread_t and returns the thread activation associated with it. SPL and Friends BSD–based and Mach–based operating systems contain legacy functions designed for basic single-processor synchronization. These include functions such as splhigh, splbio, splx, and other similar functions. Since these functions are not particularly useful for synchronization in an SMP situation, they are not particularly useful as synchronization tools in OS X. Mach Scheduling and Thread Interfaces Kernel Thread APIs 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 86If you are porting legacy code from earlier Mach–based or BSD–based operating systems, you must find an alternate means of providing synchronization. In many cases, this is as simple as taking the kernel or network funnel. In parts of the kernel, the use of spl functions does nothing, but causes no harm if you are holding a funnel (and results in a panic if you are not). In other parts of the kernel, spl macros are actually used. Because spl cannot necessarily be used for itsintended purpose, itshould not be used in general unless you are writing code it a part of the kernel that already uses it. You should instead use alternate synchronization primitives such as those described in “Synchronization Primitives” (page 128). Wait Queues and Wait Primitives The wait queue API is used extensively by the scheduler and is closely tied to the scheduler in itsimplementation. It is also used extensively in locks, semaphores, and other synchronization primitives. The wait queue API is both powerful and flexible, and as a result issomewhat large. Not all of the API is exported outside the scheduler, and parts are not useful outside the context of the wait queue functions themselves. This section documents only the public API. The wait queue API includes the following functions: void wait_queue_init(wait_queue_t wq, int policy); extern wait_queue_t wait_queue_t wait_queue_alloc(int policy); void wait_queue_free(wait_queue_t wq); void wait_queue_lock(wait_queue_t wq); void wait_queue_lock_try(wait_queue_t wq); void wait_queue_unlock(wait_queue_t wq); boolean_t wait_queue_member(wait_queue_t wq, wait_queue_sub_t wq_sub); boolean_t wait_queue_member_locked(wait_queue_t wq, wait_queue_sub_t wq_sub); kern_return_t wait_queue_link(wait_queue_t wq, wait_queue_sub_t wq_sub); kern_return_t wait_queue_unlink(wait_queue_t wq, wait_queue_sub_t wq_sub); kern_return_t wait_queue_unlink_one(wait_queue_t wq, wait_queue_sub_t *wq_subp); void wait_queue_assert_wait(wait_queue_t wq, event_t event, int interruptible); void wait_queue_assert_wait_locked(wait_queue_t wq, event_t event, int interruptible, boolean_t unlocked); kern_return_t wait_queue_wakeup_all(wait_queue_t wq, event_t event, int result); kern_return_t wait_queue_peek_locked(wait_queue_t wq, event_t event, thread_t *tp, wait_queue_t *wqp); Mach Scheduling and Thread Interfaces Kernel Thread APIs 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 87void wait_queue_pull_thread_locked(wait_queue_t wq, thread_t thread, boolean_t unlock); thread_t wait_queue_wakeup_identity_locked(wait_queue_t wq, event_t event, int result, boolean_t unlock); kern_return_t wait_queue_wakeup_one(wait_queue_t wq, event_t event, int result); kern_return_t wait_queue_wakeup_one_locked(wait_queue_t wq, event_t event, int result, boolean_t unlock); kern_return_t wait_queue_wakeup_thread(wait_queue_t wq, event_t event, thread_t thread, int result); kern_return_t wait_queue_wakeup_thread_locked(wait_queue_t wq, event_t event, thread_t thread, int result, boolean_t unlock); kern_return_t wait_queue_remove(thread_t thread); Most of the functions and their arguments are straightforward and are not presented in detail. However, a few require special attention. Most of the functions take an event_t as an argument. These can be arbitrary 32-bit values, which leads to the potential for conflicting events on certain wait queues. The traditional way to avoid this problem is to use the address of a data object that is somehow related to the code in question as that 32-bit integer value. For example, if you are waiting for an event that indicates that a new block of data has been added to a ring buffer, and if that ring buffer’s head pointer was called rb_head, you might pass the value &rb_head as the event ID. Because wait queue usage does not generally cross address space boundaries, this is generally sufficient to avoid any event ID conflicts. Notice the functions ending in _locked. These functions require that your thread be holding a lock on the wait queue before they are called. Functions ending in _locked are equivalent to their nonlocked counterparts (where applicable) except that they do not lock the queue on entry and may not unlock the queue on exit (depending on the value of unlock). The remainder of this section does not differentiate between locked and unlocked functions. The wait_queue_alloc and wait_queue_init functions take a policy parameter, which can be one of the following: ● SYNC_POLICY_FIFO—first-in, first-out ● SYNC_POLICY_FIXED_PRIORITY—policy based on thread priority Mach Scheduling and Thread Interfaces Kernel Thread APIs 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 88● SYNC_POLICY_PREPOST—keep track of number of wakeups where no thread was waiting and allow threadsto immediately continue executing without waiting until that count reaches zero. Thisisfrequently used when implementing semaphores. You should not use the wait_queue_init function outside the scheduler. Because a wait queue is an opaque object outside that context, you cannot determine the appropriate size for allocation. Thus, because the size could change in the future, you should always use wait_queue_alloc and wait_queue_free unless you are writing code within the scheduler itself. Similarly, the functions wait_queue_member, wait_queue_member_locked, wait_queue_link, wait_queue_unlink, and wait_queue_unlink_one are operations on subordinate queues, which are not exported outside the scheduler. The function wait_queue_member determines whether a subordinate queue is a member of a queue. The functions wait_queue_link and wait_queue_unlink link and unlink a given subordinate queue from its parent queue, respectively. The function wait_queue_unlink_one unlinks the first subordinate queue in a given parent and returns it. The function wait_queue_assert_wait causes the calling thread to wait on the wait queue until it is either interrupted (by a thread timer, for example) or explicitly awakened by another thread. The interruptible flag indicates whether this function should allow an asynchronous event to interrupt waiting. The function wait_queue_wakeup_all wakes up all threads waiting on a given queue for a particular event. The function wait_queue_peek_locked returns the first thread from a given wait queue that is waiting on a given event. It does not remove the thread from the queue, nor does it wake the thread. It also returns the wait queue where the thread was found. If the thread is found in a subordinate queue, other subordinate queues are unlocked, as is the parent queue. Only the queue where the thread was found remains locked. The function wait_queue_pull_thread_locked pulls a thread from the wait queue and optionally unlocks the queue. This is generally used with the result of a previous call to wait_queue_peek_locked. The function wait_queue_wakeup_identity_locked wakes up the first thread that is waiting for a given event on a given wait queue and starts it running but leaves the thread locked. It then returns a pointer to the thread. This can be used to wake the first thread in a queue and then modify unrelated structures based on which thread was actually awakened before allowing the thread to execute. The function wait_queue_wakeup_one wakes up the first thread that is waiting for a given event on a given wait queue. The function wait_queue_wakeup_thread wakes up a given thread if and only if it is waiting on the specified event and wait queue (or one of its subordinates). Mach Scheduling and Thread Interfaces Kernel Thread APIs 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 89The function wait_queue_remove wakes a given thread without regard to the wait queue or event on which it is waiting. Mach Scheduling and Thread Interfaces Kernel Thread APIs 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 90In OS X kernel programming, the term context has several meanings that appear similar on the surface, but differ subtly. First, the term context can refer to a BSD process or Mach task. Switching from one process to another is often referred to as a context switch. Second, context can refer to the part of the operating system in which your code resides. Examples of this include thread contexts, the interrupt context, the kernel context, an application’s context, a Carbon File Manager context, and so on. Even for this use of the term, the exact meaning depends, ironically, on the context in which the term is used. Finally, context can refer to a bootstrap context. In Mach, the bootstrap task is assigned responsibility for looking up requests for Mach ports. As part of this effort, each Mach task is registered in one of two groups—either in the startup context or a user’s login context. (In theory, Mach can support any number of independent contexts, however the use of additional contexts is beyond the scope of this document.) For the purposes of this chapter, the term context refers to a bootstrap context. When OS X first boots, there is only the top-level context, which is generally referred to as the startup context. All other contexts are subsets of this context. Basic system services that rely on Mach ports must be started in this context in order to work properly. When a user logs in, the bootstrap task creates a new context called the login context. Programs run by the user are started in the login context. This allows the user to run a program that provides an alternate port lookup mechanism if desired, causing that user’s tasks to get a different port when the tasks look up a basic service. This has the effect of replacing that service with a user-defined version in a way that changes what the user’s tasks see, but does not affect any of the rest of the system. To avoid wasting memory, currently the login context is destroyed when the user logs out (orshortly thereafter). This behavior may change in the future, however. In the current implementation, programs started by the user will no longer be able to look up Mach ports after logout. If a program does not need to do any port lookup, it will not be affected. Other programs will terminate, hang, or behave erratically. For example, in Mac OS 10.1 and earlier, sshd continuesto function when started from a user context. However, since it is unable to communicate with lookupd or netinfo, it stops accepting passwords. This is not a particularly useful behavior. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 91 Bootstrap ContextsOther programs such as esound, however, continue to work correctly after logout when started from a user context. Other programs behave correctly in their default configuration but fail in other configurations—for example, when authentication support is enabled. There are no hard and fast rules for which programs will continue to operate after their bootstrap context is destroyed. Only thorough testing can tell you whether any given program will misbehave if started from a user context, since even programs that do not appear to directly use Mach communication may still do so indirectly. In OS X v10.2, a great deal of effort has gone into making sure that programs that use only standard BSD services and functions do not use Mach lookups in a way that would fail if started from a user context. If you find an application that breaks when started from a Terminal.app window, please file a bug report. How Contexts Affect Users From the perspective of a user, contexts are generally unimportant as long as they do not want a program to survive past the end of their login session. Contexts do become a problem for the administrator, however. For example, if the administrator upgrades sshd by killing the old version, starting the new one, and logging out, strange things could happen since the context in which sshd was running no longer exists. Contexts also pose an issue for usersrunning background jobs with nohup or users detaching terminalsessions using screen. There are times when it is perfectly reasonable for a program to survive past logout, but by default, this does not occur. There are three basic ways that a user can get around this. In the case of daemons, they can modify the startup scripts to start the application. On restart, the application will be started in the startup context. This is not very practical if the computer in question isin heavy use, however. Fortunately, there are other waysto startservices in a startup context. The second way to run a service in the startup context is to use ssh to connect to the computer. Since sshd is running in the startup context, programs started from an ssh session also register themselves in the startup context. (Note that a user can safely kill the main sshd process without being logged out. The user just needs to be careful to kill the right one.) The third way isto log in asthe console user (>console), which causes LoginWindow to exit and causes init to spawn a getty process on the console. Since init spawns getty, which spawns login, which spawns the user’s shell, any programs started from the text console will be in the startup context. Bootstrap Contexts How Contexts Affect Users 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 92More generally, any process that is the child of a process in the startup context (other than those inherited by init because their parent process exited) is automatically in the startup context. Any process that is the child of a process in the login context is, itself, in the login context. This means that daemons can safely fork children at any time and those children will be in the startup context, as will programs started from the console (not the Console application). This also meansthat any program started by a user in a terminal window, from Finder, from the Dock, and so on, will be in the currently logged in user’s login context, even if that user runs the application using su or sudo. How Contexts Affect Developers If you are writing only kernel code, contexts are largely irrelevant (unless you are creating a new context, of course). However, kernel developers frequently need to write a program that registers itself in the startup context in order to provide some level of driver communication. For example, you could write a user-space daemon that brokers configuration information for a sound driver based on which user is logged in at the time. In the most general case, the problem ofstarting an application in the startup context can be solved by creating a startup script for your daemon, which causesit to be run in the startup context after the next reboot. However, users generally do not appreciate having to reboot their computers to install a new driver. Asking the user to connect to his or her own computer with ssh to execute a script is probably not reasonable, either. The biggest problem with forcing a reboot, of course, is that users often install several programs at once. Rebooting between each install inconveniences the end user, and has no other benefit. For that reason, you should not force the user to restart. Instead, you should offer the user the option, noting that the software may not work correctly until the user restarts. While this does not solve the fundamental problem, it does at least minimize the most common source of complaints. There are a number of ways to force a program to start in the startup context without rebooting or using ssh. However, these are not robust solutions, and are not recommended. A standard API for starting daemons is under consideration. When an official API becomes available, this chapter will be updated to discuss it. Bootstrap Contexts How Contexts Affect Developers 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 93Those of you who are already familiar with writing device drivers for Mac OS 9 or for BSD will discover that writing driversfor OS X requiressome new ways of thinking. In creating OS X, Apple has completely redesigned the Macintosh I/O architecture, providing a framework for simplified driver development that supports many categories of devices. This framework is called the I/O Kit. From a programming perspective, the I/O Kit provides an abstract view of the system hardware to the upper layers of OS X. The I/O Kit uses an object-oriented programming model, implemented in a restricted subset of C++ to promote increased code reuse. By starting with properly designed base classes, you gain a head start in writing a new driver; with much of the driver code already written, you need only to fill in the specific code that makes your driver different. For example, all SCSI controllers deliver a fairly standard set of commands to a device, but do so via different low-level mechanisms. By properly using object-oriented programming methodology, a SCSI driver can implement those low-level transport portions without reimplementing the higher level SCSI protocol code. Similar opportunities for code reuse can be found in most types of drivers. Part of the philosophy of the I/O Kit is to make the design completely open. Rather than hiding parts of the API in an attempt to protect developers from themselves, all of the I/O Kit source is available as part of Darwin. You can use the source code as an aid to designing (and debugging) new drivers. Instead of hiding the interfaces, Apple’s designers have chosen to lead by example. Sample code and classes show the recommended (easy) way to write a driver. However, you are not prevented from doing things the hard way (or the wrong way). Instead, attention has been concentrated on making the “best” ways easy to follow. Redesigning the I/O Model You might ask why Apple chose to redesign the I/O model. At first glance, it mightseem that reusing the model from Mac OS 9 or FreeBSD would have been an easier choice. There are several reasons for the decision, however. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 94 I/O Kit OverviewNeither the Mac OS 9 driver model nor the FreeBSD model offered a feature set rich enough to meet the needs of OS X. The underlying operating-system technology of OS X is very different from that of Mac OS 9. The OS X kernel is significantly more advanced than the previous Mac OS system architecture; OS X needs to handle memory protection, preemption, multiprocessing, and other features not present (orsubstantially less pervasive) in previous versions of the Mac OS. Although FreeBSD supports these features, the BSD driver model did not offer the automatic configuration, stacking, power management, or dynamic device-loading features required in a modern, consumer-oriented operating system. By redesigning the I/O architecture, Apple’s engineers can take best advantage of the operating-system features in OS X. For example, virtual memory (VM) is not a fundamental part of the operating system in Mac OS 9. Thus, every driver writer must know about (and write for) VM. This has presented certain complications for developers. In contrast, OS X has simplified driver interaction with VM. VM capability is inherent in the OS X operating system and cannot be turned off by the user. Thus, VM capabilities can be abstracted into the I/O Kit, and the code for handling VM need not be written for every driver. OS X offers an unprecedented opportunity to reuse code. In Mac OS 9, for example, all software development kits (SDKs) were independent of each other, duplicating functionality between them. In OS X, the I/O Kit is delivered as part of the basic developer tools, and code is shared among its various parts. In contrast with traditional I/O models, the reusable code model provided by the I/O Kit can decrease your development work substantially. In porting drivers from Mac OS 9, for example, the OS X counterparts have been up to 75% smaller. In general, all hardware support is provided directly by I/O Kit entities. One exception to this rule is imaging devicessuch as printers,scanners, and digital cameras(although these do make some use of I/O Kit functionality). Specifically, although communication with these devices is handled by the I/O Kit (for instance, under the FireWire or USB families), support for particular device characteristics is handled by user-space code (see “For More Information” (page 100) for further discussion). If you need to support imaging devices, you should employ the appropriate imaging software development kit (SDK). The I/O Kit attempts to represent, in software, the same hierarchy that exists in hardware. Some things are difficult to abstract, however. When the hardware hierarchy is difficult to represent (for example, if layering violations occur), then the I/O Kit abstractions provide less help for writing drivers. In addition, all drivers exist to drive hardware; all hardware is different. Even with the reusable model provided by the I/O Kit, you still need to be aware of any hardware quirks that may impact a higher-level view of the device. The code to support those quirks still needs to be unique from driver to driver. I/O Kit Overview Redesigning the I/O Model 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 95Although most developers should be able to take full advantage of I/O Kit device families (see “Families” (page 96)), there will occasionally be some who cannot. Even those developers should be able to make use of parts of the I/O Kit, however. In any case, the source code is always available. You can replace functionality and modify the classes yourself if you need to do so. In designing the I/O Kit, one goal has been to make developers’ lives easier. Unfortunately, it is not possible to make all developers’ lives uniformly easy. Therefore, a second goal of the I/O Kit design is to meet the needs of the majority of developers, without getting in the way of the minority who need lower level access to the hardware. I/O Kit Architecture The I/O Kit provides a model of system hardware in an object-oriented framework. Each type of service or device is represented by a C++ class; each discrete service or device is represented by an instance (object) of that class. There are three major conceptual elements of the I/O Kit architecture: ● “Families” (page 96) ● “Drivers” (page 97) ● “Nubs” (page 97) Families A family defines a collection of high-level abstractions common to all devices of a particular category that takes the form of C code and C++ classes. Families may include headers, libraries, sample code, test harnesses, and documentation. They provide the API, generic support code, and at least one example driver (in the documentation). Families provide services for many different categories of devices. For example, there are protocol families (such as SCSI, USB, and FireWire), storage families (disk), network families, and families to describe human interface devices (mouse and keyboard). When devices have features in common, the software that supports those features is most likely found in a family. Common abstractions are defined and implemented by the family, allowing all drivers in a family to share similar features easily. For example, all SCSI controllers have certain things they must do, such as scanning the SCSI bus. The SCSI family defines and implementsthe functionality that is common to SCSI controllers. Because thisfunctionality has been included in the SCSI family, you do not need to include scanning code (for example) in your new SCSI controller driver. I/O Kit Overview I/O Kit Architecture 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 96Instead, you can concentrate on device-specific details that make your driver different from other SCSI drivers. The use of families means there is less code for you to write. Families are dynamically loadable; they are loaded when needed and unloaded when no longer needed. Although some common families may be preloaded at system startup, all families should be considered to be dynamically loadable (and, therefore, potentially unloaded). See the “Connection Example” (page 98) for an illustration. Drivers A driver is an I/O Kit object that manages a specific device or bus, presenting a more abstract view of that device to other parts of the system. When a driver is loaded, its required families are also loaded to provide necessary, common functionality. The request to load a driver causes all of its dependent requirements (and their requirements) to be loaded first. After all requirements are met, the requested driver is loaded as well. See “Connection Example” (page 98) for an illustration. Note that families are loaded upon demand of the driver, not the other way around. Occasionally, a family may already be loaded when a driver demands it; however, you should never assume this. To ensure that all requirements are met, each device driver should list all of its requirements in its property list. Most drivers are in a client-provider relationship, wherein the driver must know about both the family from which it inherits and the family to which it connects. A SCSI controller driver, for example, must be able to communicate with both the SCSI family and the PCI family (as a client of PCI and provider of SCSI). A SCSI disk driver communicates with both the SCSI and storage families. Nubs A nub is an I/O Kit object that represents a point of connection for a driver. It represents a controllable entity such as a disk or a bus. A nub is loaded as part of the family that instantiates it. Each nub provides access to the device or service that it represents and provides services such as matching, arbitration, and power management. The concept of nubs can be more easily visualized by imagining a TV set. There is a wire attached to your wall that provides TV service from somewhere. For all practical purposes, it is permanently associated with that provider, the instantiating class (the cable company who installed the line). It can be attached to the TV to provide a service (cable TV). That wire is a nub. Each nub provides a bridge between two drivers (and, by extension, between two families). It is most common that a driver publishes one nub for each individual device or service it controls. (In this example, imagine one wire for every home serviced by the cable company.) I/O Kit Overview I/O Kit Architecture 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 97It is also possible for a driver that controls only a single device or service to act as its own nub. (Imagine the antenna on the back of your TV that has a built-in wire.) See the “Connection Example” (page 98) for an illustration of the relationship between nubs and drivers. Connection Example Figure 12-1 (page 98) illustrates the I/O Kit architecture, using several example drivers and their corresponding nubs. Note that many different driver combinations are possible; this diagram shows only one possibility. In this case, a SCSI stack is shown, with a PCI controller, a disk, and a SCSI scanner. The SCSI disk is controlled by a kernel-resident driver. The SCSI scanner is controlled by a driver that is part of a user application. Figure 12-1 I/O Kit architecture IOPCIBridge family PCI bus driver IOSCSIParallelController family SCSI card driver IOBlockStorageDriver family SCSI disk driver IOPCIDevice nubs IOSCSIParallelDevice nubs IOMedia nub Disk User application User space Kernel space Device interface User client This example illustrates how a SCSI disk driver (Storage family) is connected to the PCI bus. The connection is made in several steps. I/O Kit Overview I/O Kit Architecture 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 981. The PCI bus driver discovers a PCI device and announces its presence by creating a nub (IOPCIDevice). The nub’s class is defined by the PCI family. IOPCIBridge family PCI bus driver IOPCIDevice nubs Video card Main logic board ATA SCSI card 2. The bus driver identifies (matches) the correct device driver and requests that the driver be loaded. At the end of this matching process, a SCSI controller driver has been found and loaded. Loading the controller driver causes all required families to be loaded as well. In this case, the SCSI family is loaded; the PCI family (also required) is already present. The SCSI controller driver is given a reference to the IOPCIDevice nub. 3. The SCSI controller driver scans the SCSI bus for devices. Upon finding a device, it announces the presence of the device by creating a nub (IOSCSIDevice). The class of this nub is defined by the SCSI family. IOPCIBridge family PCI bus driver IOSCSIParallelController family SCSI card driver IOPCIDevice nubs IOSCSIParallelDevice nubs SCSI disk Unknown device SCSI scanner 1 5 6 I/O Kit Overview I/O Kit Architecture 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 994. The controller driver identifies (matches) the correct device driver and requests that the driver be loaded. At the end of this matching process, a disk driver has been found and loaded. Loading the disk driver causes all required families to be loaded as well. In this case, the Storage family is loaded; the SCSI family (also required) is already present. The disk driver is given a reference to the IOSCSIDevice nub. IOPCIBridge family PCI bus driver IOSCSIParallelController family SCSI card driver IOBlockStorageDriver family SCSI disk driver IOPCIDevice nubs IOSCSIParallelDevice nubs IOMedia nub Disk For More Information For more information on the I/O Kit, you should read the document I/O Kit Fundamentals, available from Apple’s developer documentation website, http://developer.apple.com/documentation. It provides a good general overview of the I/O Kit. In addition to I/O Kit Fundamentals, the website contains a number of HOWTO documents and topic-specific documents that describe issues specific to particular technology areas such as FireWire and USB. I/O Kit Overview For More Information 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 100The BSD portion of the OS X kernel is derived primarily from FreeBSD, a version of 4.4BSD that offers advanced networking, performance, security, and compatibility features. BSD variants in general are derived (sometimes indirectly) from 4.4BSD-Lite Release 2 from the Computer Systems Research Group (CSRG) at the University of California at Berkeley. BSD provides many advanced features, including the following: ● Preemptive multitasking with dynamic priority adjustment. Smooth and fair sharing of the computer between applications and users is ensured, even under the heaviest of loads. ● Multiuser access. Many people can use an OS X system simultaneously for a variety of things. This means, for example, thatsystem peripheralssuch as printers and disk drives are properly shared between all users on the system or the network and that individual resource limits can be placed on users or groups of users, protecting critical system resources from overuse. ● Strong TCP/IP networking with support for industry standards such as SLIP, PPP, and NFS. OS X can interoperate easily with other systems as well as act as an enterprise server, providing vital functions such as NFS (remote file access) and email services, or Internet services such as HTTP, FTP, routing, and firewall (security) services. ● Memory protection. Applications cannot interfere with each other. One application crashing does not affect others in any way. ● Virtual memory and dynamic memory allocation. Applications with large appetitesfor memory are satisfied while still maintaining interactive response to users. With the virtual memory system in OS X, each application has access to its own 4 GB memory address space; this should satisfy even the most memory-hungry applications. ● Support for kernel threads based on Mach threads. User-level threading packages are implemented on top of kernel threads. Each kernel thread is an independently scheduled entity. When a thread from a user process blocks in a system call, other threads from the same process can continue to execute on that or other processors. By default, a process in the conventional sense has one thread, the main thread. A user process can use the POSIX thread API to create other user threads. ● SMP support. Support is included for computers with multiple CPUs. ● Source code. Developers gain the greatest degree of control over the BSD programming environment because source is included. ● Many of the POSIX APIs. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 101 BSD OverviewBSD Facilities The facilities that are available to a user process are logically divided into two parts: kernel facilities and system facilities implemented by or in cooperation with a server process. The facilities implemented in the kernel define the virtual machine in which each process runs. Like many real machines, this virtual machine has memory management, an interrupt facility, timers, and counters. The virtual machine also allows access to files and other objects through a set of descriptors. Each descriptor resembles a device controller and supports a set of operations. Like devices on real machines, some of which are internal to the machine and some of which are external, parts of the descriptor machinery are built into the operating system, while other parts are often implemented in server processes. The BSD component provides the following kernel facilities: ● processes and protection ● host and process identifiers ● process creation and termination ● user and group IDs ● process groups ● memory management ● text, data, stack, and dynamic shared libraries ● mapping pages ● page protection control ● POSIX synchronization primitives ● POSIX shared memory ● signals ● signal types ● signal handlers ● sending signals ● timing and statistics ● real time ● interval time ● descriptors ● files ● pipes BSD Overview BSD Facilities 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 102● sockets ● resource controls ● process priorities ● resource utilization and resource limits ● quotas ● system operation support ● bootstrap operations ● shut-down operations ● accounting BSD system facilities (facilities that may interact with user space) include ● generic input/output operations such as read and write, nonblocking, and asynchronous operations ● file-system operations ● interprocess communication ● handling of terminals and other devices ● process control ● networking operations Differences between OS X and BSD Although the BSD portion of OS X is primarily derived from FreeBSD, some changes have been made: ● The sbrk() system call for memory management is deprecated. Its use is not recommended in OS X. ● The OS X runtime model uses a different object file format for executables and shared objects, and a different mechanism for executing some of those executables. The primary native format is Mach-O. This format is supported by the dynamic link editor (dyld). The PEF binary file format is supported by the Code Fragment Manager (CFM). The kernel supports execve() with Mach-O binaries. Mapping and management of Mach-O dynamic shared libraries, as well as launching of PEF-based applications, are performed by user-space code. ● OS X does not support memory-mapped devices through the mmap() function. (Graphic device support and other subsystems provide similar functionality, but using different APIs.) In OS X, this interface should be done through user clients. See the Apple I/O Kit documents for additional information. ● The swapon() call is not supported; macx_swapon() is the equivalent call from the Mach pager. BSD Overview Differences between OS X and BSD 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 103● The Unified Buffer Cache implementation in OS X differs from that found in FreeBSD. ● Mach provides a number of IPC primitives that are not traditionally found in UNIX. See “Boundary Crossings” (page 109) for more information on Mach IPC. Some System V primitives are supported, but their use is discouraged in favor of POSIX equivalents. ● Several changes have been made to the BSD security model to support single-user and multiple-administrator configurations, including the ability to disable ownership and permissions on a volume-by-volume basis. ● The locking mechanism used throughout the kernel differs substantially from the mechanism used in FreeBSD. ● The kernel extension mechanism used by OS X is completely different. The OS X driver layer, the I/O Kit, is an object-oriented driver stack written in C++. The general kernel programming interfaces, or KPIs, are used to write non-driver kernel extensions. These mechanisms are described more in “I/O Kit Overview” (page 94) and KPI Reference , respectively. In addition, several new features have been added that are specific to the OS X (Darwin) implementation of BSD. These features are not found in FreeBSD. ● enhancements to file-system buffer cache and file I/O clustering ● adaptive and speculative read ahead ● user-process controlled read ahead ● time aging of the file-system buffer cache ● enhancements to file-system support ● implementation of Apple extensions for ISO-9660 file systems ● multithreaded asynchronous I/O for NFS ● addition of system calls to support semantics of Mac OS Extended (HFS+) file systems ● additions to naming conventions for pathnames, as required for accessing multiple forks in Mac OS Extended file systems For Further Reading The BSD component of the OS X kernel is complex. A complete description is beyond the scope of this document. However, many excellent references exist for this component. If you are interested in BSD, be sure to refer to the bibliography for further information. BSD Overview For Further Reading 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 104Although the BSD layer of OS X is derived from 4.4BSD, keep in mind that it is not identical to 4.4BSD. Some functionality of 4.4 BSD has not been included in OS X. Some new functionality has been added. The cited reference materials are recommended for additional reading. However, they should not be presumed as forming a definitive description of OS X. BSD Overview For Further Reading 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 105OS X provides“out-of-the-box”support forseveral different file systems. These include Mac OS Extended format (HFS+), the BSD standard file system format (UFS), NFS (an industry standard for networked file systems), ISO 9660 (used for CD-ROM), MS-DOS, SMB (Windows file sharing standard), AFP (Mac OS file sharing), and UDF. Support is also included for reading the older, Mac OS Standard format (HFS) file-system type; however, you should not plan to format new volumes using Mac OS Standard format. OS X cannot boot from these file systems, nor does the Mac OS Standard format provide some of the information required by OS X. The Mac OS Extended format provides many of the same characteristics as Mac OS Standard format but adds additional support for modern features such as file permissions, longer filenames, Unicode, both hard and symbolic links, and larger disk sizes. UFS provides case sensitivity and other characteristics that may be expected by BSD commands. In contrast, Mac OS Extended Format is not case-sensitive (but is case-preserving). OS X currently can boot and “root” from an HFS+, UFS, ISO, NFS, or UDF volume. That is, OS X can boot from and mount a volume of any of these types and use it as the primary, or root, file system. Other file systems can also be mounted, allowing usersto gain accessto additional volume formats and features. NFS provides access to network servers as if they were locally mounted file systems. The Carbon application environment mimics many expected behaviors of Mac OS Extended format on top of both UFS and NFS. These include such characteristics as Finder Info, file ID access, and aliases. By using the OS X Virtual File System (VFS) capability and writing kernel extensions, you can add support for other file systems. Examples of file systems that are not currently supported in OS X but that you may wish to add to the system include the Andrew file system (AFS) and the Reiser file system (ReiserFS). If you want to support a new volume format or networking protocol, you’ll need to write a file-system kernel extension. Working With the File System In OS X, the vnode structure providesthe internal representation of a file or directory (folder). There is a unique vnode allocated for each active file or folder, including the root. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 106 File Systems OverviewWithin a file system, operations on specific files and directories are implemented via vnodes and VOP (vnode operation) calls. VOP calls are used for operations on individual files or directories (such as open, close, read, or write). Examples include VOP_OPEN to open a file and VOP_READ to read file contents. In contrast, file-system–wide operations are implemented using VFS calls. VFS calls are primarily used for operations on entire file systems; examples include VFS_MOUNT and VFS_UNMOUNT to mount or unmount a file system, respectively. File-system writers need to provide stubs for each of these sets of calls. VFS Transition The details of the VFS subsystem in OS X are in the process of changing in order to make the VFS interface sustainable. If you are writing a leaf file system, these changes will still affect you in many ways. please contact Apple Developer Support for more information. File Systems Overview VFS Transition 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 107OS X kernel extensions (KEXTs) provide mechanisms to extend and modify the networking infrastructure of OS X dynamically, without recompiling or relinking the kernel. The effect is immediate and does not require rebooting the system. Networking KEXTs can be used to ● monitor network traffic ● modify network traffic ● receive notification of asynchronous events from the driver layer In the last case, such events are received by the data link and network layers. Examples of these events include power management events and interface status changes. Specifically, KEXTs allow you to ● create protocol stacks that can be loaded and unloaded dynamically and configured automatically ● create modulesthat can be loaded and unloaded dynamically atspecific positionsin the network hierarchy. The Kernel Extension Manager dynamically adds KEXTs to the running OS X kernel inside the kernel’s address space. An installed and enabled network-related KEXT is invoked automatically, depending on its position in the sequence of protocol components, to process an incoming or outgoing packet. All KEXTs provide initialization and termination routines that the Kernel Extension Manager invokes when it loads or unloads the KEXT. The initialization routine handles any operations that are needed to complete the incorporation of the KEXT into the kernel, such as updating protosw and domain structures (through programmatic interfaces). Similarly, the termination routine must remove references to the NKE from these structures to unload itself successfully. NKEs must provide a mechanism, such as a reference count, to ensure that the NKE can terminate without leaving dangling pointers. For additional information on the networking portions of the OS X kernel, you should read the document Network Kernel Extensions Programming Guide . 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 108 Network ArchitectureTwo applications can communicate in a number of ways—for example, by using pipes or sockets. The applicationsthemselves are unaware of the underlying mechanismsthat provide this communication. However this communication occurs by sending data from one program into the kernel, which then sends the data to the second program. As a kernel programmer, it is your job to create the underlying mechanisms responsible for communication between your kernel code and applications. This communication is known as crossing the user-kernel boundary. This chapter explains various ways of crossing that boundary. In a protected memory environment, each process is given its own address space. This means that no program can modify another program’s data unless that data also resides in its own memory space (shared memory). The same applies to the kernel. It resides in its own address space. When a program communicates with the kernel, data cannot simply be passed from one address space to the other as you might between threads (or between programs in environments like Mac OS 9 and most real-time operating systems, which do not have protected memory). We refer to the kernel’s address space as kernel space, and collectively refer to applications’ address spaces as user space. For this reason, applications are also commonly referred to as user-space programs, or user programs for short. When the kernel needs a small amount of data from an application, the kernel cannot just dereference a pointer passed in from that application, since that pointer is relative to the application’s address space. Instead, the kernel generally copies that information into storage within its own address space. When a large region of data needs to be moved, it may map entire pages into kernel space for efficiency. The same behavior can be seen in reverse when moving data from the kernel to an application. Because it is difficult to move data back and forth between the kernel and an application, this separation is called a boundary. It isinherently time consuming to copy data, even if that data isjust the user-space address of a shared region. Thus, there is a performance penalty whenever a data exchange occurs. If this penalty is a serious problem, it may affect which method you choose for crossing the user-kernel boundary. Also, by trying to minimize the number of boundary crossings, you may find ways to improve the overall design of your code. This is particularly significant if your code is involved in communication between two applications, since the user-kernel boundary must be crossed twice in that case. There are a number of ways to cross the user-kernel boundary. Some of them are covered in this chapter in the following sections: 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 109 Boundary Crossings● “Mach Messaging and Mach Interprocess Communication (IPC)” (page 112) ● “BSD syscall API ” (page 116) ● “BSD ioctl API” (page 116) ● “BSD sysctl API ” (page 117) ● “Memory Mapping and Block Copying” (page 125) In addition, the I/O Kit uses the user-client/device-interface API for most communication. Because that API is specific to the I/O Kit, it is not covered in this chapter. The user client API is covered in I/O Kit Fundamentals, Accessing Hardware From Applications, and I/O Kit Device Driver Design Guidelines. The ioctl API is also specific to the construction of device drivers, and is largely beyond the scope of this document. However, since ioctl is a BSD API, it is covered at a glance for your convenience. This chapter covers one subset of Mach IPC—the Mach remote procedure call (RPC) API. It also covers the syscall, sysctl, memory mapping, and block copying APIs. Security Considerations Crossing the user-kernel boundary represents a security risk if the kernel code operates on the data in any substantial way (beyond writing it to disk or passing it to another application). You must carefully perform bounds checking on any data passed in, and you must also make sure your code does not dereference memory that no longer belongs to the client application. Also, under no circumstances should you run unverified program code passed in from user space within the kernel. See “Security Considerations” (page 24) for further information. Choosing a Boundary Crossing Method The first step in setting up user-kernel data exchange is choosing a means to do that exchange. First, you must consider the purpose for the communication. Some crucial factors are latency, bandwidth, and the kernel subsystem involved. Before choosing a method of communication, however, you should first understand at a high-level each of these forms of communication. Mach messaging and Mach interprocess communication (IPC) are relatively low-level ways of communicating between two Mach tasks (processes), as well as between a Mach task and the kernel. These form the basis for most communication outside of BSD and the I/O Kit. The Mach remote procedure call (RPC) API is a high level procedural abstraction built on top of Mach IPC. Mach RPC is the most common use of IPC. Boundary Crossings Security Considerations 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 110The BSD syscall API is an API for calling kernel functions from user space. It is used extensively when writing file systems and networking protocols, in ways that are very subsystem-dependent. Developers are strongly discouraged from using the syscall API outside of file-system and network extensions, as no plug-in API exists for registering a new system call with the syscall mechanism. The BSD sysctl API (in its revised form) supersedes the syscall API and also provides a relatively painless way to change individual kernel variablesfrom userspace. It has a straightforward plug-in architecture, making it a good choice where possible. Memory mapping and block copying are used in conjunction with one of the other APIs mentioned, and provide ways of moving large amounts of data (more than a few bytes) or variably sized data to and from kernel space. Kernel Subsystems The choice of boundary crossing methods depends largely on the part of the kernel into which you are adding code. In particular, the boundary crossing method preferred for the I/O Kit is different from that preferred for BSD, which is different from that preferred for Mach. If you are writing a device driver or other related code, you are probably dealing with the I/O Kit. In that case, you should instead read appropriate sections in I/O Kit Fundamentals, Accessing Hardware From Applications, and I/O Kit Device Driver Design Guidelines. If you are writing code that resides in the BSD subsystem (for example, a file system), you should generally use BSD APIs such as syscall or sysctl unless you require high bandwidth or exceptionally low latency. If you are writing code that resides anywhere else, you will probably have to use Mach messaging. Bandwidth and Latency The guidelines in the previous section apply to most communication between applications and kernel code. The methods mentioned, however, are somewhat lacking where high bandwidth or low latency are concerns. If you require high bandwidth, but latency is not an issue, you should probably consider doing memory-mapped communication. For large messagesthisis handled somewhat transparently by Mach RPC, making it a reasonable choice. For BSD portions of the kernel, however, you must explicitly pass pointers and use copyin and copyout to move large quantities of data. Thisis discussed in more detail in “Memory Mapping and Block Copying” (page 125). Boundary Crossings Choosing a Boundary Crossing Method 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 111If you require low latency but bandwidth is not an issue, sysctl and syscall are not good choices. Mach RPC, however, may be an acceptable solution. Another possibility is to actually wire a page of memory (see “Memory Mapping and Block Copying” (page 125) for details),start an asynchronous Mach RPC simpleroutine (to process the data), and use either locks or high/low water marks (buffer fullness) to determine when to read and write data. This can work for high-bandwidth communication as well. If you require both high bandwidth and low latency, you should also look at the user client/device interface model used in the I/O Kit, since that model has similar requirements. Mach Messaging and Mach Interprocess Communication (IPC) Mach IPC and Mach messaging are the basis for much of the communication in OS X. In many cases, however, these facilities are used indirectly by services implemented on top of one of them. Mach messaging and IPC are fundamentally similar except that Mach messaging is stateless, which prevents certain types of error recovery, as explained later. Except where explicitly stated, this section treats the two as equivalent. The fundamental unit of Mach IPC isthe port. The concept of Mach ports can be difficult to explain in isolation, so instead this section assumes a passing knowledge of a similar concept, that of ports in TCP/IP. In TCP/IP, a server listens for incoming connections over a network on a particular port. Multiple clients can connect to the port and send and receive data in word-sized or multiple-word–sized blocks. However, only one server process can be bound to the port at a time. In Mach IPC, the concept is the same, but the players are different. Instead of multiple hosts connecting to a TCP/IP port, you have multiple Mach tasks on the same computer connecting to a Mach port. Instead of firewall rules on a port, you have port rights that specify what tasks can send data to a particular Mach port. Also, TCP/IP ports are bidirectional, while Mach ports are unidirectional, much like UNIX pipes. This means that when a Mach task connects to a port, it generally allocates a reply port and sends a message containing send rights to that reply port so that the receiving task can send messages back to the sending task. As with TCP/IP, multiple client tasks can open connections to a Mach port, but only one task can be listening on that port at a time. Unlike TCP/IP, however, the IPC mechanism itself provides an easy means for one task to hand off the right to listen to an arbitrary task. The term receive rights refers to a task’s ability to listen on a given port. Receive rights can be sent from task to task in a Mach message. In the case of Mach IPC (but not Mach messaging), receive rights can even be configured to automatically return to the original task if the new task crashes or becomes unreachable (for example, if the new task isrunning on another computer and a router crashes). Boundary Crossings Mach Messaging and Mach Interprocess Communication (IPC) 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 112In addition to specifying receive rights, Mach ports can specify which tasks have the right to send data. A task with send rights may be able to send once, or may be able to arbitrarily send data to a given port, depending on the nature of the rights. Using Well-Defined Ports Before you can use Mach IPC for task communication, the sending task must be able to obtain send rights on the receiving task’s task port. Historically, there are several ways of doing this, not all of which are supported by OS X. For example, in OS X, unlike most other Mach derivatives, there is no service server or name server. Instead, the bootstrap task and mach_init subsume this functionality. When a task is created, it is given send rights to a bootstrap port for sending messages to the bootstrap task. Normally a task would use this port to send a message that gives the bootstrap task send rights on another port so that the bootstrap task can then return data to the calling task. Various routines exist in bootstrap.h that abstract this process. Indeed, most users of Mach IPC or Mach messaging actually use Mach remote procedure calls (RPC), which are implemented on top of Mach IPC. Since direct use of IPC is rarely desirable (because it is not easy to do correctly), and because the underlying IPC implementation has historically changed on a regular basis, the details are not covered here. You can find more information on using Mach IPC directly in the Mach 3 Server Writer’s Guide from Silicomp (formerly the Open Group, formerly the Open Software Foundation Research Institute), which can be obtained from the developer section of Apple’s website. While much of the information contained in that book is not fully up-to-date with respect to OS X, it should still be a relatively good resource on using Mach IPC. Remote Procedure Calls (RPC) Mach RPC is the most common use for Mach IPC. It is frequently used for user-kernel communication, but can also be used for task to task or even computer-to-computer communication. Programmers frequently use Mach RPC for setting certain kernel parameters such as a given thread’s scheduling policy. RPC is convenient because it is relatively transparent to the programmer. Instead of writing long, complex functionsthat handle ports directly, you have only to write the function to be called and a small RPC definition to describe how to export the function as an RPC interface. After that, any application with appropriate permissions can call those functions as if they were local functions, and the compiler will convert them to RPC calls. In the directory osfmk/mach (relative to your checkout of the xnu module from CVS), there are a number of files ending in .defs; these files contain the RPC definitions. When the kernel (or a kernel module) is compiled, the Mach Interface Generator(MIG) usesthese definitionsto create IPC code to support the functions exported via RPC. Normally, if you want to add a new remote procedure call, you should do so by adding a definition to one of these existing files. (See “Building and Debugging Kernels” (page 155) for more information on obtaining kernel sources.) Boundary Crossings Mach Messaging and Mach Interprocess Communication (IPC) 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 113What follows is an example of the definition for a routine, one of the more common uses of RPC. routine thread_policy_get( thread : thread_act_t; flavor : thread_policy_flavor_t; out policy_info : thread_policy_t, CountInOut; inout get_default : boolean_t); Notice the C-like syntax of the definition. Each parameter in the routine roughly maps onto a parameter in the C function. The C prototype for this function follows. kern_return_t thread_policy_get( thread_act_t act, thread_policy_flavor_t flavor, thread_policy_t policy_info, mach_msg_type_number_t *count, boolean_t get_default); The first two parameters are integers, and are passed as call-by-value. The third is a struct containing integers. It is an outgoing parameter, which means that the values stored in that variable will not be received by the function, but will be overwritten on return. Note: The parameters are all word-sized or multiples of the word size. Smaller data are impossible because of limitations inherent to the underlying Mach IPC mechanisms. From there it becomes more interesting. The fourth parameter in the C prototype is a representation of the size of the third. In the definition file, this is represented by an added option, CountInOut. The MIG option CountInOut specifies that there is to be an inout parameter called count. An inout parameter is one in which the original value can be read by the function being called, and its value is replaced on return from that function. Unlike a separate inout parameter, however, the value initially passed through this parameter is not directly set by the calling function. Instead, it is tied to the policy_info parameter so that the number of integers in policy_info is transparently passed in through this parameter. Boundary Crossings Mach Messaging and Mach Interprocess Communication (IPC) 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 114In the function itself, the function checks the count parameter to verify that the buffer size is at least the size of the data to be returned to prevent exceeding array bounds. The function changes the value stored in count to be the desired size and returns an error if the buffer is not large enough (unless the buffer pointer is null, in which case it returns success). Otherwise, it dereferences the various fields of the policy_info parameter and in so doing, stores appropriate values into it, then returns. Note: Since Mach RPC is done via message passing, inout parameters are technically call-by-value-return and not call-by-reference. For more realistic call-by-reference, you need to pass a pointer. The distinction is not particularly significant except when aliasing occurs. (Aliasing means having a single variable visible in the same scope under two or more different names.) In addition to the routine, Mach RPC also has a simpleroutine. A simpleroutine is a routine that is, by definition, asynchronous. It can have no out or inout parameters and no return value. The caller does not wait for the function to return. One possible use for this might be to tell an I/O device to send data as soon as it is ready. In that use, the simpleroutine might simply wait for data, then send a message to the calling task to indicate the availability of data. Another important feature of MIG is that of the subsystem. In MIG, a subsystem is a group of routines and simpleroutines that are related in some way. For example, the semaphore subsystem contains related routinesthat operate on semaphores. There are also subsystemsfor varioustimers, parts of the virtual memory (VM) system, and dozens of others in various places throughout the kernel. Most of the time, if you need to use RPC, you will be doing it within an existing subsystem. The details of creating a new subsystem are beyond the scope of this document. Developers needing to add a new Mach subsystem should consult the Mach 3 ServerWriter’s Guide from The Open Group (TOG), which can be obtained from various locations on the internet. Another feature of MIG is the type. A type in MIG is exactly the same thing as it is in programming languages. However, the construction of aggregate types differs somewhat. type clock_flavor_t = int; type clock_attr_t = array[*:1] of int; type mach_timespec_t = struct[2] of int; Data of type array is passed as the user-space address of what is assumed to be a contiguous array of the base type, while a struct is passed by copying all of the individual values of an array of the base type. Otherwise, these are treated similarly. A “struct” is not like a C struct, as elements of a MIG struct must all be of the same base type. Boundary Crossings Mach Messaging and Mach Interprocess Communication (IPC) 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 115The declaration syntax issimilar to Pascal, where *:1 and 2 representsizesfor the array orstructure, respectively. The *:1 construct indicates a variable-sized array, where the size can be up to 1, inclusive, but no larger. Calling RPC From User Applications RPC, as mentioned previously, is virtually transparent to the client. The procedure call looks like any other C function call, and no additional library linkage is needed. You need only to bring the appropriate headers in with a #include directive. The compiler automatically recognizes the call as a remote procedure call and handles the underlying MIG aspects for you. BSD syscall API The syscall API is the traditional UNIX way of calling kernel functions from user space. Its implementation variesfrom one part of the kernel to the next, however, and it is completely unsupported for loadable modules. For this reason, it is not a recommended way of getting data into or out of the kernel in OS X unless you are writing a file system. File systems have to support a number of standard system calls (for example, mount), but do so by means of generic file system routinesthat call the appropriate file-system functions. Thus, if you are writing a file system, you need to implement those functions, but you do not need to write the code that handles the system calls directly. For more information on implementing syscall support in file systems,see the chapter “File Systems Overview” (page 106). BSD ioctl API The ioctl interface provides a way for an application to send certain commands or information to a device driver. These can be used for parameter tuning (though this is more commonly done with sysctl), but can also be used for sending instructions for the driver to perform a particular task (for example, rewinding a tape drive). The use of the ioctl interface is essentially the same under OS X as it is in other BSD-derived operating systems, except in the way that device drivers register themselves with the system. In OS X, unlike most BSDs, the contents of the /dev directory are created dynamically by the kernel. This file system mounted on /dev is referred to as devfs. You can, of course, still manually create device nodes with mknod, because devfs is union mounted over the root file system. Boundary Crossings BSD syscall API 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 116The I/O Kit automatically registers some types of devices with devfs, creating a node in /dev. If your device family does not do that, you can manually register yourself in devfs using cdevsw_add or bdevsw_add (for character and block devices, respectively). When registering a device manually with devfs, you create a struct cdevsw or struct bdevsw yourself. In that device structure, one of the function pointers is to an ioctl function. You must define the particular values passed to the ioctl function in a header file accessible to the person compiling the application. A user application can also look up the device using the I/O Kit function call getMatchingServices and then use various I/O Kit calls to tune parameter instead. For more information on looking up a device driver from an application, see the document Accessing Hardware From Applications. You can also find additional information about writing an ioctl in The Design and Implementation of the 4.4 BSD Operating System. See the bibliography at the end of this document for more information. BSD sysctl API The system control (sysctl) API is specifically designed for kernel parameter tuning. This functionality supersedesthe syscall API, and also provides an easy way to tune simple kernel parameters without actually needing to write a handler routine in the kernel. The sysctl namespace is divided into several broad categories corresponding to the purpose of the parameters in it. Some of these areas include ● kern—general kernel parameters ● vm—virtual memory options ● fs—filesystem options ● machdep—machine dependent settings ● net—network stack settings ● debug—debugging settings ● hw—hardware parameters (generally read-only) ● user—parameters affecting user programs ● ddb—kernel debugger Most of the time, programs use the sysctl call to retrieve the current value of a kernel parameter. For example, in OS X, the hw sysctl group includesthe option ncpu, which returnsthe number of processorsin the current computer (or the maximum number of processors supported by the kernel on that particular computer, whichever is less). Boundary Crossings BSD sysctl API 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 117The sysctl API can also be used to modify parameters (though most parameters can only be changed by the root). For example, in the net hierarchy, net.inet.ip.forwarding can be set to 1 or 0, to indicate whether the computer should forward packets between multiple interfaces (basic routing). General Information on Adding a sysctl When adding a sysctl, you must do all of the following first: ● add the following includes: #include #include #include #include ● add -no-cpp-precomp to your compiler options in Project Builder (or to CFLAGS in your makefile if building by hand). Adding a sysctl Procedure Call Adding a system control (sysctl) was once a daunting task requiring changes to dozens of files. With the current implementation, a system control can be added simply by writing the appropriate handler functions and then registering the handler with the system at runtime. The old-style sysctl, which used fixed numbers for each control, is deprecated. Note: Because this is largely a construct of the BSD subsystem, all path names in this section can be assumed to be from /path/to/xnu-version/bsd/. Also, you may safely assume that all program code snippets should go into the main source file for your subsystem or module unless otherwise noted, and that in the case of modules, function calls should be made from your start or stop routines unless otherwise noted. The preferred way of adding a sysctl looks something like the following: SYSCTL_PROC(_hw, OID_AUTO, l2cr, CTLTYPE_INT|CTLFLAG_RW, &L2CR, 0, &sysctl_l2cr, "I", "L2 Cache Register"); Boundary Crossings BSD sysctl API 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 118The _PROC part indicates that you are registering a procedure to provide the value (as opposed to simply reading from a static address in kernel memory). _hw is the top level category (in this case, hardware), and OID_AUTO indicates that you should be assigned the next available control ID in that category (as opposed to the old-style, fixed ID controls). l2cr is the name of your control, which will be used by applications to look up the number of your control using sysctlbyname. Note: Not all top level categories will necessarily accept the addition of a user-specified new-style sysctl. If you run into problems, you should try a different top-level category. CTLTYPE_INT indicates that the value being changed is an integer. Other legal values are CTLTYPE_NODE, CTLTYPE_STRING, CTLTYPE_QUAD, and CTLTYPE_OPAQUE (also known as CTLTYPE_STRUCT). CTLTYPE_NODE isthe only one that isn’tsomewhat obvious. It refersto a node in the sysctl hierarchy that isn’t directly usable, but instead is a parent to other entries. Two examples of nodes are hw and kern. CTLFLAG_RW indicatesthat the value can be read and written.Other legal values are CTLFLAG_RD, CTLFLAG_WR, CTLFLAG_ANYBODY, and CTLFLAG_SECURE. CTLFLAG_ANYBODY means that the value should be modifiable by anybody. (The default is for variables to be changeable only by root.) CTLFLAG_SECURE means that the variable can be changed only when running at securelevel <= 0 (effectively, in single-user mode). L2CR is the location where the sysctl will store its data. Since the address is set at compile time, however, this must be a global variable or a static local variable. In this case, L2CR is a global of type unsigned int. The number 0 is a second argument that is passed to your function. This can be used, for example, to identify which sysctl was used to call your handler function if the same handler function is used for more than one control. In the case of strings, this is used to store the maximum allowable length for incoming values. sysctl_l2cr is the handler function for this sysctl. The prototype for these functions is of the form static int sysctl_l2cr SYSCTL_HANDLER_ARGS; If the sysctl is writable, the function may either use sysctl_handle_int to obtain the value passed in from user space and store it in the default location or use the SYSCTL_IN macro to store it into an alternate buffer. This function must also use the SYSCTL_OUT macro to return a value to user space. "I" indicates that the argument should refer to a variable of type integer (or a constant, pointer, or other piece of data of equivalent width), as opposed to "L" for a long, "A" for a string, "N" for a node (a sysctl that is the parent of a sysctl category or subcategory), or "S" for a struct. "L2 Cache Register" is a human-readable description of your sysctl. In order for a control to be accessible from an application, it must be registered. To do this, you do the following: Boundary Crossings BSD sysctl API 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 119sysctl_register_oid(&sysctl__hw_l2cr); You should generally do this in an init routine for a loadable module. If your code is not part of a loadable module, you should add your sysctl to the list of built-in OIDs in the file kern/sysctl_init.c. If you study the SYSCTL_PROC constructor macro, you will notice that sysctl__hw_l2cr is the name of a variable created by that macro. This meansthat the SYSCTL_PROC line must be before sysctl_register_oid in the file, and must be in the same (or broader) scope. This name is in the form of sysctl_ followed by the name of it’s parent node, followed by another underscore ( _ ) followed by the name of your sysctl. A similar function, sysctl_unregister_oid exists to remove a sysctl from the registry. If you are writing a loadable module, you should be certain to do this when your module is unloaded. In addition to registering your handler function, you also have to write the function. The following is a typical example static int myhandler SYSCTL_HANDLER_ARGS { int error, retval; error = sysctl_handle_int(oidp, oidp->oid_arg1, oidp->oid_arg2, req); if (!error && req->newptr) { /* We have a new value stored in the standard location.*/ /* Do with it as you see fit here. */ printf("sysctl_test: stored %d\n", SCTEST); } else if (req->newptr) { /* Something was wrong with the write request */ /* Do something here if you feel like it.... */ } else { /* Read request. Always return 763, just for grins. */ printf("sysctl_test: read %d\n", SCTEST); retval=763; error=SYSCTL_OUT(req, &retval, sizeof retval); } /* In any case, return success or return the reason for failure */ return error; Boundary Crossings BSD sysctl API 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 120} This demonstrates the use of SYSCTL_OUT to send an arbitrary value out to user space from the sysctl handler. The “phantom” req argument is part of the function prototype when the SYSCTL_HANDLER_ARGS macro is expanded, as is the oidp variable used elsewhere. The remaining arguments are a pointer (type indifferent) and the length of data to copy (in bytes). This code sample also introduces a new function, sysctl_handle_int, which takes the arguments passed to the sysctl, and writes the integer into the usual storage area (L2CR in the earlier example, SCTEST in this one). If you want to see the new value without storing it (to do a sanity check, for example), you should instead use the SYSCTL_IN macro, whose arguments are the same as SYSCTL_OUT. Registering a New Top Level sysctl In addition to adding new sysctl options, you can also add a new category or subcategory. The macro SYSCTL_DECL can be used to declare a node that can have children. This requires modifying one additional file to create the child list. For example, if your main C file does this: SYSCTL_DECL(_net_newcat); SYSCTL_NODE(_net, OID_AUTO, newcat, CTLFLAG_RW, handler, "new category"); then this is basically the same thing as declaring extern sysctl_oid_list sysctl__net_newcat_children in your program. In order for the kernel to compile, or the module to link, you must then add this line: struct sysctl_oid_list sysctl__net_newcat_children; If you are not writing a module, this should go in the file kern/kern_newsysctl.c. Otherwise, it should go in one of the files of your module. Once you have created this variable, you can use _net_newcat as the parent when creating a new control. As with any sysctl, the node (sysctl__net_newcat) must be registered with sysctl_register_oid and can be unregistered with sysctl_unregister_oid. Boundary Crossings BSD sysctl API 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 121Note: When creating a top level sysctl, parent is simply left blank, for example, SYSCTL_NODE( , OID_AUTO, _topname, flags, handler_fn, "desc"); Adding a Simple sysctl If your sysctl only needsto read a value out of a variable, then you do not need to write a function to provide access to that variable. Instead, you can use one of the following macros: ● SYSCTL_INT(parent, nbr, name, access, ptr, val, descr) ● SYSCTL_LONG(parent, nbr, name, access, ptr, descr) ● SYSCTL_STRING(parent, nbr, name, access, arg, len, descr) ● SYSCTL_OPAQUE(parent, nbr, name, access, ptr, len, descr) ● SYSCTL_STRUCT(parent, nbr, name, access, arg, type, descr) The first four parameters for each macro are the same as for SYSCTL_PROC (described in the previous section) as is the last parameter. The len parameter (where applicable) gives a length of the string or opaque object in bytes. The arg parameters are pointersjust like the ptr parameters. However, the parameters named ptr are explicitly described as pointers because you must explicitly use the “address of” (&) operator unless you are already working with a pointer. Parameters called arg either operate on base types that are implicitly pointers or add the & operator in the appropriate place during macro expansion. In both cases, the argument should refer to the integer, character, or other object that the sysctl will use to store the current value. The type parameter is the name of the type minus the “struct”. For example, if you have an object of type struct scsipi, then you would use scsipi as that argument. The SYSCTL_STRUCT macro is functionally equivalent to SYSCTL_OPAQUE, except that it hides the use of sizeof. Finally, the val parameter for SYSCTL_INT is a default value. If the value passed in ptr is NULL, this value is returned when the sysctl is used. You can use this, for example, when adding a sysctl that is specific to certain hardware or certain compile options. One possible example of this might be a special value for feature.version that means “not present.” If that feature became available (for example, if a module were loaded by some user action), it could then update that pointer. If that module were subsequently unloaded, it could set the pointer back to NULL. Boundary Crossings BSD sysctl API 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 122Calling a sysctl From User Space Unlike RPC, sysctl requires explicit intervention on the part of the programmer. To complicate thingsfurther, there are two different ways of calling sysctl functions, and neither one worksfor every control. The old-style sysctl call can only invoke a control if it is listed in a static OID table in the kernel. The new-style sysctlbyname call will work for any user-added sysctl, but not for those listed in the static table. Occasionally, you will even find a control that isregistered in both ways, and thus available to both calls. In order to understand the distinction, you must first consider the functions used. The sysctlbyname System Call If you are calling a sysctl that was added using the new sysctl method (including any sysctl that you may have added), then your sysctl does not have a fixed number that identifies it, since it was added dynamically to the system. Since there is no approved way to get this number from user space, and since the underlying implementation is not guaranteed to remain the same in future releases, you cannot call a dynamically added control using the sysctl function. Instead, you must use sysctlbyname. sysctlbyname(char *name, void *oldp, size_t *oldlenp, void *newp, u_int newlen) The parameter name is the name of the sysctl, encoded as a standard C string. The parameter oldp is a pointer to a buffer where the old value will be stored. The oldlenp parameter is a pointer to an integer-sized buffer that holds the current size of the oldp buffer. If the oldp buffer is not large enough to hold the returned data, the call will fail with errno set to ENOMEM, and the value pointed to by oldlenp will be changed to indicate the buffer size needed for a future call to succeed. Here is an example for reading an integer, in this case a buffer size. int get_debug_bufsize() { char *name="debug.bpf_bufsize"; int bufsize, retval; size_t len; len=4; retval=sysctlbyname(name, &bufsize, &len, NULL, 0); /* Check retval here */ return bufsize; } Boundary Crossings BSD sysctl API 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 123The sysctl System Call The sysctlbyname system call is the recommended way to call system calls. However, not every built-in system control is registered in the kernel in such a way that it can be called with sysctlbyname. For this reason, you should also be aware of the sysctl system call. Note: If you are adding a sysctl, it will be accessible using sysctlbyname. You should use this system call only if the sysctl you need cannot be retrieved using sysctlbyname. In particular, you should not assume that future versions of sysctl will be backed by traditional numeric OIDs except for the existing legacy OIDs, which will be retained for compatibility reasons. The sysctl system call is part of the original historical BSD implementation of system controls. You should not depend on its use for any control that you might add to the system. The classic usage of sysctl looks like the following sysctl(int *name, u_int namelen, void *oldp, size_t *oldlenp, void *newp, u_int newlen) System controls, in this form, are based on the MIB, or Management Information Base architecture. A MIB is a list of objects and identifiers for those objects. Each object identifier, or OID, is a list of integers that represent a tokenization of a path through the sysctl tree. For example, if the hw class of sysctl is number 3, the first integer in the OID would be the number 3. If the l2cr option is built into the system and assigned the number 75, then the second integer in the OID would be 75. To put it another way, each number in the OID is an index into a node’s list of children. Here is a short example of a call to get the bus speed of the current computer: int get_bus_speed() { int mib[2], busspeed, retval; unsigned int miblen; size_t len; mib[0]=CTL_HW; mib[1]=HW_BUS_FREQ; miblen=2; len=4; retval=sysctl(mib, miblen, &busspeed, &len, NULL, 0); Boundary Crossings BSD sysctl API 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 124/* Check retval here */ return busspeed; } For more information on the sysctl system call, see the manual page sysctl. Memory Mapping and Block Copying Memory mapping is one of the more common means of communicating between two applications or between an application and the kernel. While occasionally used by itself, it is usually used in conjunction with one of the other means of boundary crossing. One way of using memory mapping is known as shared memory. In this form, one or more pages of memory are mapped into the address space of two processes. Either process can then access or modify the data stored in those shared pages. This is useful when moving large quantities of data between processes, as it allows direct communication without multiple user-kernel boundary crossings. Thus, when moving large amounts of data between processes, this is preferable to traditional message passing. The same holds true with memory mapping between an application and the kernel. The BSD sysctl and syscall interfaces (and to an extent, Mach IPC) were designed to transfer small units of data of known size, such as an array of four integers. In this regard, they are much like a traditional C function call. If you need to pass a large amount of data to a function in C, you should pass a pointer. This is also true when passing data between an application and the kernel, with the addition of memory mapping or copying to allow that pointer to be dereferenced in the kernel. There are a number of limitations to the way that memory mapping can be used to exchange data between an application and the kernel. For one, memory allocated in the kernel cannot be written to by applications, including those running as root (unless the kernel is running in an insecure mode, such as single user mode). For this reason, if a buffer must be modified by an application, the buffer must be allocated by that program, not by the kernel. When you use memory mapping for passing data to the kernel, the application allocates a block of memory and fillsit with data. It then performs a system call that passesthe addressto the appropriate function in kernel space. It should be noted, however, that the address being passed is a virtual address, not a physical address, and more importantly, it is relative to the address space of the program, which is not the same as the address space of the kernel. Since the address is a user-space virtual address, the kernel must call special functions to copy the block of memory into a kernel buffer or to map the block of memory into the kernel’s address space. Boundary Crossings Memory Mapping and Block Copying 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 125In the OS X kernel, data is most easily copied into kernel space with the BSD copyin function, and back out to user space with the copyout function. For large blocks of data, entire pages will be memory mapped using copy-on-write. For this reason, it is generally not useful to do memory mapping by hand. Getting data from the kernel to an application can be done in a number of ways. The most common method is the reverse of the above, in which the application passes in a buffer pointer, the kernel scribbles on a chunk of data, uses copyout to copy the buffer data into the address space of the application, and returns KERN_SUCCESS. Note that this is really using the buffer allocated in the application, even though the physical memory may have actually been allocated by the kernel. Assuming the kernel frees its reference to the buffer, no memory is wasted. A special case of memory mapping occurs when doing I/O to a device from user space. Since I/O operations can, in some cases, be performed by DMA hardware that operates based on physical addressing, it is vital that the memory associated with I/O buffers not be paged out while the hardware is copying data to or from the buffer. For this reason, when a driver or other kernel entity needs a buffer for I/O, it must take steps to mark it as not pageable. This step is referred to as wiring the pages in memory. Wiring pages into memory can also be helpful where high bandwidth, low latency communication is desired, as it prevents shared buffers from being paged out to disk. In general, however, this sort of workaround should be unnecessary, and is considered to be bad programming practice. Pages can be wired in two ways. When a memory region is allocated, it may be allocated in a nonpageable fashion. The details of allocating memory for I/O differ, depending on what part of the kernel you are modifying. This is described in more detail in the appropriate sections of this document, or in the case of the I/O Kit, in the API reference documentation (available from the developer section of Apple’s web site). Alternately, individual pages may be wired after allocation. The recommended way to do this is through a call to vm_wire in BSD parts of the kernel, with mlock from applications (but only by processes running as root), or with IOMemoryDescriptor::prepare in the I/O Kit. Because this can fail for a number of reasons, it is particularly crucial to check return values when wiring memory. The vm_wire call and other virtual memory topics are discussed in more detail in “Memory and Virtual Memory” (page 61). The IOMemoryDescriptor class is described in more detail in the I/O Kit API reference available from the developer section of Apple’s web site. Boundary Crossings Memory Mapping and Block Copying 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 126Summary Crossing the user-kernel boundary is not a trivial task. Many mechanisms exist for this communication, and each one has specific advantages and disadvantages, depending on the environment and bandwidth requirements. Security is a constant concern to prevent inadvertently allowing one program to access data or files from another program or user. It is every kernel programmer’s personal responsibility to take security into account any time that data crosses the user-kernel boundary. Boundary Crossings Summary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 127This chapter is not intended as an introduction to synchronization. It is assumed that you have some understanding of the basic concepts of locks and semaphores already. If you need additional background reading,synchronization is covered in most introductory operating systemstexts. However,since synchronization in the kernel is somewhat different from locking in an application this chapter does provide a brief overview to help ease the transition, or for experienced kernel developers, to refresh your memory. As an OS X kernel programmer, you have many choices of synchronization mechanisms at your disposal. The kernel itself provides two such mechanisms: locks and semaphores. A lock is used for basic protection of shared resources. Multiple threads can attempt to acquire a lock, but only one thread can actually hold it at any given time (at least for traditional locks—more on this later). While that thread holds the lock, the other threads must wait. There are several different types of locks, differing mainly in what threads do while waiting to acquire them. A semaphore is much like a lock, except that a finite number of threads can hold itsimultaneously. Semaphores can be thought of as being much like piles of tokens. Multiple threads can take these tokens, but when there are none left, a thread must wait until another thread returns one. It is important to note that semaphores can be implemented in many different ways,so Mach semaphores may not behave in the same way assemaphores on other platforms. In addition to locks and semaphores, certain low-level synchronization primitives like test and set are also available, along with a number of other atomic operations. These additional operations are described in libkern/gen/OSAtomicOperations.c in the kernelsources. Such atomic operations may be helpful if you do not need something asrobust as a full-fledged lock orsemaphore. Since they are not generalsynchronization mechanisms, however, they are beyond the scope of this chapter. Semaphores Semaphores and locks are similar, except that with semaphores, more than one thread can be doing a given operation at once. Semaphores are commonly used when protecting multiple indistinct resources. For example, you might use a semaphore to prevent a queue from overflowing its bounds. OS X uses traditional counting semaphores rather than binary semaphores (which are essentially locks). Mach semaphores obey Mesa semantics—that is, when a thread is awakened by a semaphore becoming available, it is not executed immediately. This presents the potential for starvation in multiprocessor situations when the 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 128 Synchronization Primitivessystem is under low overall load because other threads could keep downing the semaphore before the just-woken thread gets a chance to run. This is something that you should consider carefully when writing applications with semaphores. Semaphores can be used any place where mutexes can occur. This precludes their use in interrupt handlers or within the context of the scheduler, and makes it strongly discouraged in the VM system. The public API for semaphores is divided between the MIG–generated task.h file (located in your build output directory, included with #include ) and osfmk/mach/semaphore.h (included with #include ). The public semaphore API includes the following functions: kern_return_t semaphore_create(task_t task, semaphore_t *semaphore, int policy, int value) kern_return_t semaphore_signal(semaphore_t semaphore) kern_return_t semaphore_signal_all(semaphore_t semaphore) kern_return_t semaphore_wait(semaphore_t semaphore) kern_return_t semaphore_destroy(task_t task, semaphore_t semaphore) kern_return_t semaphore_signal_thread(semaphore_t semaphore, thread_act_t thread_act) which are described in or xnu/osfmk/mach/semaphore.h (except for create and destroy, which are described in . The use of these functions is relatively straightforward with the exception of the semaphore_create, semaphore_destroy, and semaphore_signal_thread calls. The value and semaphore parametersfor semaphore_create are exactly what you would expect—a pointer to the semaphore structure to be filled out and the initial value for the semaphore, respectively. The task parameter refers to the primary Mach task that will “own” the lock. This task should be the one that is ultimately responsible for the subsequent destruction of the semaphore. The task parameter used when calling semaphore_destroy must match the one used when it was created. For communication within the kernel, the task parameter should be the result of a call to current_task. For synchronization with a user process, you need to determine the underlying Mach task for that process by calling current_task on the kernel side and mach_task_self on the application side. task_t current_task(void); // returns the kernel task port task_t mach_task_self(void);// returns the task port of the current thread Synchronization Primitives Semaphores 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 129Note: In the kernel, be sure to always use current_task. In the kernel, mach_task_self returns a pointer to the kernel’s VM map, which is probably not what you want. The details of user-kernel synchronization are beyond the scope of this document. The policy parameter is passed asthe policy for the wait queue contained within the semaphore. The possible values are defined in osfmk/mach/sync_policy.h. Current possible values are: ● SYNC_POLICY_FIFO ● SYNC_POLICY_FIXED_PRIORITY ● SYNC_POLICY_PREPOST The FIFO policy is, asthe name suggests, first-in-first-out. The fixed priority policy causes wait queue reordering based on fixed thread priority policies. The prepost policy causes the semaphore_signal function to not increment the counter if no threads are waiting on the queue. This policy is needed for creating condition variables (where a thread is expected to always wait until signalled). See the section “Wait Queues and Wait Primitives” (page 87) for more information. The semaphore_signal_thread call takes a particular thread from the wait queue and places it back into one of the scheduler’s wait-queues, thus making that thread available to be scheduled for execution. If thread_act is NULL, the first thread in the queue is similarly made runnable. With the exception of semaphore_create and semaphore_destroy, these functions can also be called from user space via RPC. See “Calling RPC From User Applications” (page 116) for more information. Condition Variables The BSD portion of OS X provides msleep, wakeup, and wakeup_one, which are equivalent to condition variables with the addition of an optional time-out. You can find these functions in sys/proc.h in the Kernel framework headers. msleep(void *channel, lck_mtx_t *mtx, int priority, const char *wmesg, struct timespec *timeout); msleep0(vvoid *channel, lck_mtx_t *mtx, int priority, const char *wmesg, uint64_t deadline); wakeup(void *channel); wakeup_one(void *channel); Synchronization Primitives Condition Variables 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 130The msleep call is similar to a condition variable. It puts a thread to sleep until wakeup or wakeup_one is called on that channel. Unlike a condition variable, however, you can set a timeout measured in clock ticks. This means that it is both a synchronization call and a delay. The prototypes follow: msleep(void *channel, lck_mtx_t *mtx, int priority, const char *wmesg, struct timespec *timeout); msleep0(vvoid *channel, lck_mtx_t *mtx, int priority, const char *wmesg, uint64_t deadline); wakeup(void *channel); wakeup_one(void *channel); The three sleep calls are similar except in the mechanism used for timeouts. The function msleep0 is not recommended for general use. In these functions, channel is a unique identifier representing a single condition upon which you are waiting. Normally, when msleep is used, you are waiting for a change to occur in a data structure. In such cases, it is common to use the address of that data structure as the value for channel, as this ensures that no code elsewhere in the system will be using the same value. The priority argument has three effects. First, when wakeup is called, threads are inserted in the scheduling queue at this priority. Second, if the bit (priority & PCATCH) is set, msleep0 does not allow signals to interrupt the sleep. Third, if the bit (priority & PDROP) is zero, msleep0 drops the mutex on sleep and reacquires it upon waking. If (priority & PDROP) is one, msleep0 drops the mutex if it has to sleep, but does not reacquire it. The subsystem argument is a short text string that represents the subsystem that is waiting on this channel. This is used solely for debugging purposes. The timeout argument is used to set a maximum wait time. The thread may wake sooner, however, if wakeup or wakeup_one is called on the appropriate channel. It may also wake sooner if a signal isreceived, depending on the value of priority. In the case of msleep0, this is given as a mach abstime deadline. In the case of msleep, this is given in relative time (seconds and nanoseconds). Outside the BSD portion of the kernel, condition variables may be implemented using semaphores. Synchronization Primitives Condition Variables 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 131Locks OS X (and Mach in general) has three basic types of locks: spinlocks, mutexes, and read-write locks. Each of these has different uses and different problems. There are also many other types of locks that are not implemented in OS X, such as spin-sleep locks, some of which may be useful to implement for performance comparison purposes. Spinlocks A spinlock is the simplest type of lock. In a system with a test-and-set instruction or the equivalent, the code looks something like this: while (test_and_set(bit) != 0); In other words, until the lock is available, it simply “spins” in a tight loop that keeps checking the lock until the thread’s time quantum expires and the next thread begins to execute. Since the entire time quantum for the first thread must complete before the next thread can execute and (possibly) release the lock, a spinlock is very wasteful of CPU time, and should be used only in places where a mutex cannot be used, such as in a hardware exception handler or low-level interrupt handler. Note that a thread may not block while holding a spinlock, because that could cause deadlock. Further, preemption is disabled on a given processor while a spinlock is held. There are three basic types of spinlocks available in OS X: lck_spin_t (which supersedes simple_lock_t), usimple_lock_t, and hw_lock_t. You are strongly encouraged to not use hw_lock_t; it is only mentioned for the sake of completeness. Of these, only lck_spin_t is accessible from kernel extensions. The u in usimple stands for uniprocessor, because they are the only spinlocks that provide actual locking on uniprocessorsystems. Traditionalsimple locks, by contrast, disable preemption but do notspin on uniprocessor systems. Note that in most contexts, it is not useful to spin on a uniprocessor system, and thus you usually only need simple locks. Use of usimple locks is permissible for synchronization between thread context and interrupt context or between a uniprocessor and an intelligent device. However, in most cases, a mutex is a better choice. Important: Simple and usimple locks that could potentially be shared between interrupt context and thread context must have their use coordinated with spl (see glossary). The IPL (interrupt priority level) must always be the same when acquiring the lock, otherwise deadlock may result. (This is not an issue for kernel extensions, however, as the spl functions cannot be used there.) The spinlock functions accessible to kernel extensions consist of the following: Synchronization Primitives Locks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 132extern lck_spin_t *lck_spin_alloc_init( lck_grp_t *grp, lck_attr_t *attr); extern void lck_spin_init( lck_spin_t *lck, lck_grp_t *grp, lck_attr_t *attr); extern void lck_spin_lock( lck_spin_t *lck); extern void lck_spin_unlock( lck_spin_t *lck); extern void lck_spin_destroy( lck_spin_t *lck, lck_grp_t *grp); extern void lck_spin_free( lck_spin_t *lck, lck_grp_t *grp); extern wait_result_t lck_spin_sleep( lck_spin_t *lck, lck_sleep_action_t lck_sleep_action, event_t event, wait_interrupt_t interruptible); extern wait_result_t lck_spin_sleep_deadline( lck_spin_t *lck, lck_sleep_action_t lck_sleep_action, event_t event, wait_interrupt_t interruptible, uint64_t deadline); Synchronization Primitives Locks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 133Prototypes for these locks can be found in . The arguments to these functions are described in detail in “Using Lock Functions” (page 139). Mutexes A mutex, mutex lock, or sleep lock, is similar to a spinlock, except that instead of constantly polling, it places itself on a queue of threads waiting for the lock, then yields the remainder of its time quantum. It does not execute again until the thread holding the lock wakesit (or in some userspace variations, until an asynchronous signal arrives). Mutexes are more efficient than spinlocksfor most purposes. However, they are less efficient in multiprocessing environments where the expected lock-holding time is relatively short. If the average time is relatively short but occasionally long, spin/sleep locks may be a better choice. Although OS X does not support spin/sleep locksin the kernel, they can be easily implemented on top of existing locking primitives. If your code performance improves as a result of using such locks, however, you should probably look for ways to restructure your code, such as using more than one lock or moving to read-write locks, depending on the nature of the code in question. See “Spin/Sleep Locks” (page 138) for more information. Because mutexes are based on blocking, they can only be used in places where blocking is allowed. For this reason, mutexes cannot be used in the context of interrupt handlers. Interrupt handlers are not allowed to block because interrupts are disabled for the duration of an interrupt handler, and thus, if an interrupt handler blocked, it would prevent the scheduler from receiving timer interrupts, which would prevent any other thread from executing, resulting in deadlock. For a similar reason, it is not reasonable to block within the scheduler. Also, blocking within the VM system can easily lead to deadlock if the lock you are waiting for is held by a task that is paged out. However, unlike simple locks, it is permissible to block while holding a mutex. This would occur, for example, if you took one lock, then tried to take another, but the second lock was being held by another thread. However, this is generally not recommended unless you carefully scrutinize all uses of that mutex for possible circular waits, as it can result in deadlock. You can avoid this by always taking locks in a certain order. In general, blocking while holding a mutex specific to your code isfine aslong as you wrote your code correctly, but blocking while holding a more global mutex is probably not, since you may not be able to guarantee that other developers’ code obeys the same ordering rules. A Mach mutex is of type mutex_t. The functions that operate on mutexes include: lck_mtx_t *lck_mtx_alloc_init(lck_grp_t *grp, lck_attr_t *attr); extern void lck_mtx_init( lck_mtx_t *lck, Synchronization Primitives Locks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 134lck_grp_t *grp, lck_attr_t *attr); extern void lck_mtx_lock( lck_mtx_t *lck); extern void lck_mtx_unlock( lck_mtx_t *lck); extern void lck_mtx_destroy(lck_mtx_t *lck, lck_grp_t *grp); extern void lck_mtx_free( lck_mtx_t *lck, lck_grp_t *grp); extern wait_result_tlck_mtx_sleep( lck_mtx_t *lck, lck_sleep_action_t lck_sleep_action, event_t event, wait_interrupt_t interruptible); extern wait_result_tlck_mtx_sleep_deadline( lck_mtx_t *lck, lck_sleep_action_t lck_sleep_action, event_t event, wait_interrupt_t interruptible, uint64_t deadline); extern void lck_mtx_assert( lck_mtx_t *lck, unsigned int type); as described in . The arguments to these functions are described in detail in “Using Lock Functions” (page 139). Synchronization Primitives Locks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 135Read-Write Locks Read-write locks (also called shared-exclusive locks) are somewhat different from traditional locks in that they are not always exclusive locks. A read-write lock is useful when shared data can be reasonably read concurrently by multiple threads except while a thread is modifying the data. Read-write locks can dramatically improve performance if the majority of operations on the shared data are in the form of reads(since it allows concurrency), while having negligible impact in the case of multiple writes. A read-write lock allows this sharing by enforcing the following constraints: ● Multiple readers can hold the lock at any time. ● Only one writer can hold the lock at any given time. ● A writer must block until all readers have released the lock before obtaining the lock for writing. ● Readers arriving while a writer is waiting to acquire the lock will block until after the writer has obtained and released the lock. The first constraint allows read sharing. The second constraint prevents write sharing. The third prevents read-write sharing, and the fourth prevents starvation of the writer by a steady stream of incoming readers. Mach read-write locks also provide the ability for a reader to become a writer and vice-versa. In locking terminology, an upgrade is when a reader becomes a writer, and a downgrade is when a writer becomes a reader. To prevent deadlock, some additional constraints must be added for upgrades and downgrades: ● Upgrades are favored over writers. ● The second and subsequent concurrent upgrades will fail, causing that thread’s read lock to be released. The first constraint is necessary because the reader requesting an upgrade is holding a read lock, and the writer would not be able to obtain a write lock until the reader releases its read lock. In this case, the reader and writer would wait for each other forever. The second constraint is necessary to prevents the deadlock that would occur if two readers wait for the other to release its read lock so that an upgrade can occur. The functions that operate on read-write locks are: extern lck_rw_t *lck_rw_alloc_init( lck_grp_t *grp, lck_attr_t *attr); extern void lck_rw_init( lck_rw_t *lck, lck_grp_t *grp, Synchronization Primitives Locks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 136lck_attr_t *attr); extern void lck_rw_lock( lck_rw_t *lck, lck_rw_type_t lck_rw_type); extern void lck_rw_unlock( lck_rw_t *lck, lck_rw_type_t lck_rw_type); extern void lck_rw_lock_shared( lck_rw_t *lck); extern void lck_rw_unlock_shared( lck_rw_t *lck); extern void lck_rw_lock_exclusive( lck_rw_t *lck); extern void lck_rw_unlock_exclusive( lck_rw_t *lck); extern void lck_rw_destroy( lck_rw_t *lck, lck_grp_t *grp); extern void lck_rw_free( lck_rw_t *lck, lck_grp_t *grp); extern wait_result_t lck_rw_sleep( lck_rw_t *lck, lck_sleep_action_t lck_sleep_action, Synchronization Primitives Locks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 137event_t event, wait_interrupt_t interruptible); extern wait_result_t lck_rw_sleep_deadline( lck_rw_t *lck, lck_sleep_action_t lck_sleep_action, event_t event, wait_interrupt_t interruptible, uint64_t deadline); This is a more complex interface than that of the other locking mechanisms, and actually is the interface upon which the other locks are built. The functions lck_rw_lock and lck_rw_lock lock and unlock a lock as either shared (read) or exclusive (write), depending on the value of lck_rw_type., which can contain either LCK_RW_TYPE_SHARED or LCK_RW_TYPE_EXCLUSIVE. You should always be careful when using these functions, as unlocking a lock held in shared mode using an exclusive call or vice-versa will lead to undefined results. The arguments to these functions are described in detail in “Using Lock Functions” (page 139). Spin/Sleep Locks Spin/sleep locks are not implemented in the OS X kernel. However, they can be easily implemented on top of existing locks if desired. For short waits on multiprocessor systems, the amount of time spent in the context switch can be greater than the amount of time spent spinning. When the time spent spinning while waiting for the lock becomes greater than the context switch overhead, however, mutexes become more efficient. For this reason, if there is a large degree of variation in wait time on a highly contended lock, spin/sleep locks may be more efficient than traditional spinlocks or mutexes. Ideally, a program should be written in such a way that the time spent holding a lock is always about the same, and the choice of locking is clear. However, in some cases, this is not practical for a highly contended lock. In those cases, you may consider using spin/sleep locks. Synchronization Primitives Locks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 138The basic principle ofspin/sleep locksissimple. A thread takesthe lock if it is available. If the lock is not available, the thread may enter a spin cycle. After a certain period of time (usually a fraction of a time quantum or a small number of time quanta), the spin routine’s time-out is reached, and it returns failure. At that point, the lock places the waiting thread on a queue and puts it to sleep. In other variations on this design, spin/sleep locks determine whether to spin or sleep according to whether the lock-holding thread is currently on another processor (or is about to be). For short wait periods on multiprocessor computers, the spin/sleep lock is more efficient than a mutex, and roughly as efficient as a standard spinlock. For longer wait periods, the spin/sleep lock is significantly more efficient than the spinlock and only slightly less efficient than a mutex. There is a period near the transition between spinning and sleeping in which the spin/sleep lock may behave significantly worse than either of the basic lock types, however. Thus, spin/sleep locks should not be used unless a lock is heavily contended and has widely varying hold times. When possible, you should rewrite the code to avoid such designs. Using Lock Functions While most of the locking functions are straightforward, there are a few detailsrelated to allocating, deallocating, and sleeping on locks that require additional explanation. As the syntax of these functions is identical across all of the lock types, this section explains only the usage for spinlocks. Extending this to other lock types is left as a (trivial) exercise for the reader. The first thing you must do when allocating locks is to allocate a lock group and a lock attribute set. Lock groups are used to name locks for debugging purposes and to group locks by function for general understandability. Lock attribute sets allow you to set flags that alter the behavior of a lock. The following code illustrates how to allocate an attribute structure and a lock group structure for a lock. In this case, a spinlock is used, but with the exception of the lock allocation itself, the process is the same for other lock types. Listing 17-1 Allocating lock attributes and groups (lifted liberally from kern_time.c) lck_grp_attr_t *tz_slock_grp_attr; lck_grp_t *tz_slock_grp; lck_attr_t *tz_slock_attr; lck_spin_t *tz_slock; /* allocate lock group attribute and group */ tz_slock_grp_attr = lck_grp_attr_alloc_init(); lck_grp_attr_setstat(tz_slock_grp_attr); Synchronization Primitives Locks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 139tz_slock_grp = lck_grp_alloc_init("tzlock", tz_slock_grp_attr); /* Allocate lock attribute */ tz_slock_attr = lck_attr_alloc_init(); //lck_attr_setdebug(tz_slock_attr); // set the debug flag //lck_attr_setdefault(tz_slock_attr); // clear the debug flag /* Allocate the spin lock */ tz_slock = lck_spin_alloc_init(tz_slock_grp, tz_slock_attr); The first argument to the lock initializer, of type lck_grp_t, is a lock group. This is used for debugging purposes, including lock contention profiling. The details of lock tracing are beyond the scope of this document, however, every lock must belong to a group (even if that group contains only one lock). The second argument to the lock initializer, of type lck_attr_t, contains attributes for the lock. Currently, the only attribute available islock debugging. This attribute can be set using lck_attr_setdebug and cleared with lck_attr_setdefault. To dispose of a lock, you simply call the matching free functions. For example: lck_spin_free(tz_slock, tz_slock_grp); lck_attr_free(tz_slock_attr); lck_grp_free(tz_slock_grp); lck_grp_attr_free(tz_slock_grp_attr); Note: While you can safely dispose of the lock attribute and lock group attribute structures, it is important to keep track of the lock group associated with a lock as long as the lock exists, since you will need to pass the group to the lock's matching free function when you deallocate the lock (generally at unload time). The other two interesting functions are lck_spin_sleep and lck_spin_sleep_deadline. These functions release a spinlock and sleep until an event occurs, then wake. The latter includes a timeout, at which point it will wake even if the event has not occurred. extern wait_result_t lck_spin_sleep( lck_rspin_t *lck, Synchronization Primitives Locks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 140lck_sleep_action_t lck_sleep_action, event_t event, wait_interrupt_t interruptible); extern wait_result_t lck_spin_sleep_deadline( lck_spin_t *lck, lck_sleep_action_t lck_sleep_action, event_t event, wait_interrupt_t interruptible, uint64_t deadline); The parameter lck_sleep_action controls whether the lock will be reclaimed after sleeping prior to this function returning. The valid options are: LCK_SLEEP_DEFAULT Release the lock while waiting for the event, then reclaim it. Read-write locks are held in the same mode as they were originally held. LCK_SLEEP_UNLOCK Release the lock and return with the lock unheld. LCK_SLEEP_SHARED Reclaim the lock in shared mode (read-write locks only). LCK_SLEEP_EXCLUSIVE Reclaim the lock in exclusive mode (read-write locks only). The event parameter can be any arbitrary integer, but it must be unique across the system. To ensure uniqueness, a common programming practice isto use the address of a global variable (often the one containing a lock) as the event value. For more information on these events, see “Event and Timer Waits” (page 143). The parameter interruptible indicates whether the scheduler should allow the wait to be interrupted by asynchronous signals. If this is false, any false wakes will result in the process going immediately back to sleep (with the exception of a timer expiration signal, which will still wake lck_spin_sleep_deadline). Synchronization Primitives Locks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 141This chapter containsinformation about miscellaneousservices provided by the OS X kernel. For most projects, you will probably never need to use most of these services, but if you do, you will find it hard to do without them. This chapter containsthese sections:“Using Kernel Time Abstractions” (page 142),“Boot Option Handling” (page 146), “Queues” (page 147), and “Installing Shutdown Hooks” (page 148). Using Kernel Time Abstractions There are two basic groups of time abstractionsin the kernel. One group includesfunctionsthat provide delays and timed wake-ups. The other group includesfunctions and variablesthat provide the current wall clock time, the time used by a given process, and other similar information. This section describes both aspects of time from the perspective of the kernel. Obtaining Time Information There are a number of ways to get basic time information from within the kernel. The officially approved methods are those that Mach exports in kern/clock.h. These include the following: void clock_get_uptime(uint64_t *result); void clock_get_system_microtime( uint32_t *secs, uint32_t *microsecs); void clock_get_system_nanotime( uint32_t *secs, uint32_t *nanosecs); void clock_get_calendar_microtime( uint32_t *secs, uint32_t *microsecs); void clock_get_calendar_nanotime( uint32_t *secs, uint32_t *nanosecs); 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 142 Miscellaneous Kernel ServicesThe function clock_get_uptime returns a value in AbsoluteTime units. For more information on using AbsoluteTime, see “Using Mach Absolute Time Functions” (page 144). The functions clock_get_system_microtime and clock_get_system_nanotime return 32-bit integers containing seconds and microseconds or nanoseconds, respectively, representing the system uptime. The functions clock_get_calendar_microtime and clock_get_calendar_nanotime return 32-bit integers containing seconds and microseconds or nanoseconds, respectively, representing the current calendar date and time since the epoch (January 1, 1970). In some parts of the kernel, you may find other functions that return type mach_timespec_t. This type is similar to the traditional BSD struct timespec, except that fractions of a second are measured in nanoseconds instead of microseconds: struct mach_timespec { unsigned int tv_sec; clock_res_t tv_nsec; }; typedef struct mach_timespec *mach_timespec_t; In addition to the traditional Mach functions, if you are writing code in BSD portions of the kernel you can also get the current calendar (wall clock) time as a BSD timeval, as well as find out the calendar time when the system was booted by doing the following: #include struct timeval tv=time; /* calendar time */ struct timeval tv_boot=boottime; /* calendar time when booting occurred */ For other information, you should use the Mach functions listed previously. Event and Timer Waits Each part of the OS X kernel has a distinct API for waiting a certain period of time. In most cases, you can call these functions from other parts of the kernel. The I/O Kit provides IODelay and IOSleep. Mach provides functions based on AbsoluteTime, as well as a few based on microseconds. BSD provides msleep. Miscellaneous Kernel Services Using Kernel Time Abstractions 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 143Using IODelay and IOSleep IODelay, provided by the I/O Kit, abstracts a timed spin. If you are delaying for a short period of time, and if you need to be guaranteed that your wait will not be stopped prematurely by delivery of asynchronous events, this is probably the best choice. If you need to delay for several seconds, however, this is a bad choice, because the CPU that executes the wait will spin until the time has elapsed, unable to handle any other processing. IOSleep puts the currently executing thread to sleep for a certain period of time. There is no guarantee that your thread will execute after that period of time, nor isthere a guarantee that your thread will not be awakened by some other event before the time has expired. It is roughly equivalent to the sleep call from user space in this regard. The use of IODelay and IOSleep are straightforward. Their prototypes are: IODelay(unsigned microseconds); IOSleep(unsigned milliseconds); Note the differing units. It is not practical to put a thread to sleep for periods measured in microseconds, and spinning for several milliseconds is also inappropriate. Using Mach Absolute Time Functions The following Mach time functions are commonly used. Several others are described in osfmk/kern/clock.h. Note: These are not the same functions as those listed in kern/clock.h in the Kernel framework. These functions are not exposed to kernel extensions, and are only for use within the kernel itself. void delay(uint64_t microseconds); void clock_delay_until(uint64_t deadline); void clock_absolutetime_interval_to_deadline(uint64_t abstime, uint64_t *result); void nanoseconds_to_absolutetime(uint64_t nanoseconds, uint64_t *result); void absolutetime_to_nanoseconds(uint64_t abstime, uint64_t *result); These functions are generally straightforward. However, a few points deserve explanation. Unless specifically stated, all times, deadlines, and so on, are measured in abstime units. The abstime unit is equal to the length of one bus cycle,so the duration is dependent on the busspeed of the computer. For thisreason, Mach provides conversion routines between abstime units and nanoseconds. Miscellaneous Kernel Services Using Kernel Time Abstractions 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 144Many time functions, however, provide time in seconds with nanosecond remainder. In this case, some conversion is necessary. For example, to obtain the current time as a mach abstime value, you might do the following: uint32_t secpart; uint32_t nsecpart; uint64_t nsec, abstime; clock_get_calendar_nanotime(&secpart, &nsecpart); nsec = nsecpart + (1000000000ULL * secpart); //convert seconds to nanoseconds. nanoseconds_to_absolutetime(nsec, &abstime); The abstime value is now stored in the variable abstime. Using msleep In addition to Mach and I/O Kit routines, BSD provides msleep, which is the recommended way to delay in the BSD portions of the kernel. In other parts of the kernel, you should either use wait_queue functions or use assert_wait and thread_wakeup functions, both of which are closely tied to the Mach scheduler, and are described in “Kernel Thread APIs” (page 85). Because this function is more commonly used for waiting on events, it is described further in “Condition Variables” (page 130). Handling Version Dependencies Many time-related functions such as clock_get_uptime changed as a result of the transition to KPIs in OS X v.10.4. While these changes result in a cleaner interface, this can prove challenging if you need to make a kernel extension that needs to obtain time information across multiple versions of OS X in a kernel extension that would otherwise have no version dependencies (such as an I/O Kit KEXT). Here is a list of time-related functions that are available in both pre-KPI and KPI versions of OS X: uint64_t mach_absolute_time(void); Declared In: Dependency: com.apple.kernel.mach This function returns a Mach absolute time value for the current wall clock time in units of uint64_t. Miscellaneous Kernel Services Using Kernel Time Abstractions 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 145void microtime(struct timeval *tv); Declared In: Dependency: com.apple.kernel.bsd This function returns a timeval struct containing the current wall clock time. void microuptime(struct timeval *tv); Declared In: Dependency: com.apple.kernel.bsd This function returns a timeval struct containing the current uptime. void nanotime(struct timespec *ts); Declared In: Dependency: com.apple.kernel.bsd This function returns a timespec struct containing the current wall clock time. void nanouptime(struct timespec *ts); Declared In: Dependency: com.apple.kernel.bsd This function returns a timespec struct containing the current uptime. Note: The structure declarationsfor struct timeval and struct timespec differ between 10.3 and 10.4 in their use of int, int32_t, and long data types. However, because the structure packing for the underlying data types is identical in the 32-bit world, these structures are assignment compatible. In addition to these APIs, the functionality marked __APPLE_API_UNSTABLE in was adopted as-is in OS X v.10.4 and is no longer marked unstable. Boot Option Handling OS X provides a simple parse routine, PE_parse_boot_arg, for basic boot argument passing. It supports both flags and numerical value assignment. For obtaining values, you write code similar to the following: unsigned int argval; if (PE_parse_boot_arg("argflag", &argval)) { Miscellaneous Kernel Services Boot Option Handling 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 146/* check for reasonable value */ if (argval < 10 || argval > 37) argval = 37; } else { /* use default value */ argval = 37; } Since PE_parse_boot_arg returns a nonzero value if the flag exists, you can check for the presence of a flag by using a flag that starts with a dash (-) and ignoring the value stored in argvalue. The PE_parse_boot_arg function can also be used to get a string argument. To do this, you must pass in the address of an array of type char as the second argument. The behavior of PE_parse_boot_arg is undefined if a string is passed in for a numeric variable or vice versa. Its behavior is also undefined if a string exceeds the storage space allocated. Be sure to allow enough space for the largest reasonable string including a null delimiter. No attempt is made at bounds checking, since an overflow is generally a fatal error and should reasonably prevent booting. Queues As part of its BSD infrastructure, the OS X kernel provides a number of basic support macrosto simplify handling of linked lists and queues. These are implemented as C macros, and assume a standard C struct. As such, they are probably not suited for writing code in C++. The basic types of lists and queues included are ● SLIST, a singly linked list ● STAILQ, a singly linked tail queue ● LIST, a doubly linked list ● TAILQ, a doubly linked tail queue SLIST is ideal for creating stacks or for handling large sets of data with few or no removals. Arbitrary removal, however, requires an O(n) traversal of the list. STAILQ is similar to SLIST except that it maintains pointers to both ends of the queue. This makes it ideal for simple FIFO queues by adding entries at the tail and fetching entries from the head. Like SLIST, it is inefficient to remove arbitrary elements. Miscellaneous Kernel Services Queues 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 147LIST is a doubly linked version of SLIST. The extra pointersrequire additionalspace, but allow O(1) (constant time) removal of arbitrary elements and bidirectional traversal. TAILQ is a doubly linked version of STAILQ. Like LIST, the extra pointers require additional space, but allow O(1) (constant time) removal of arbitrary elements and bidirectional traversal. Because their functionality is relatively simple, their use is equally straightforward. These macros can be found in xnu/bsd/sys/queue.h. Installing Shutdown Hooks Although OS X does not have traditional BSD-style shutdown hooks, the I/O Kit provides equivalent functionality in recent versions. Since the I/O Kit provides this functionality, you must call it from C++ code. To register for notification, you call registerSleepWakeInterest (described in IOKit/RootDomain.h) and register for sleep notification. If the system is about to be shut down, your handler is called with the message type kIOMessageSystemWillPowerOff. If the system is about to reboot, your handler gets the message type kIOMessageSystemWillRestart. If the system is about to reboot, your handler gets the message type kIOMessageSystemWillSleep. If you no longer need to receive notification (for example, if your KEXT gets unloaded), be certain to release the notifier with IONofitier::release to avoid a kernel panic on shutdown. For example, the following sample KEXT registersforsleep notifications, then logs a message with IOLog when a sleep notification occurs: #include #include #include #include #include #define ALLOW_SLEEP 1 IONotifier *notifier; extern "C" { Miscellaneous Kernel Services Installing Shutdown Hooks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 148IOReturn mySleepHandler( void * target, void * refCon, UInt32 messageType, IOService * provider, void * messageArgument, vm_size_t argSize ) { IOLog("Got sleep/wake notice. Message type was %d\n", messageType); #if ALLOW_SLEEP acknowledgeSleepWakeNotification(refCon); #else vetoSleepWakeNotification(refCon); #endif return 0; } kern_return_t sleepkext_start (kmod_info_t * ki, void * d) { void *myself = NULL; // Would pass the self pointer here if in a class instance notifier = registerPrioritySleepWakeInterest( &mySleepHandler, myself, NULL); return KERN_SUCCESS; } kern_return_t sleepkext_stop (kmod_info_t * ki, void * d) { notifier->remove(); return KERN_SUCCESS; } } // extern "C" Miscellaneous Kernel Services Installing Shutdown Hooks 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 149As discussed in the chapter “Kernel Architecture Overview” (page 14), OS X provides a kernel extension mechanism as a means of allowing dynamic loading of code into the kernel, without the need to recompile or relink. Because these kernel extensions (KEXTs) provide both modularity and dynamic loadability, they are a natural choice for any relatively self-contained service that requires access to internal kernel interfaces. Because KEXTs run in supervisor mode in the kernel’s address space, they are also harder to write and debug than user-level modules, and must conform to strict guidelines. Further, kernel resources are wired (permanently resident in memory) and are thus more costly to use than resources in a user-space task of equivalent functionality. In addition, although memory protection keeps applications from crashing the system, no such safeguards are in place inside the kernel. A badly behaved kernel extension in OS X can cause as much trouble as a badly behaved application or extension could in Mac OS 9. Bugs in KEXTs can have far more severe consequences than bugs in user-level code. For example, a memory access error in a user application can, at worst, cause that application to crash. In contrast, a memory access error in a KEXT causes a kernel panic, crashing the operating system. Finally, for security reasons, some customers restrict or don’t permit the use of third-party KEXTs. As a result, use of KEXTs is strongly discouraged in situations where user-level solutions are feasible. OS X guarantees that threading in applications is just as efficient as threading inside the kernel, so efficiency should not be an issue. Unless your application requireslow-level accessto kernel interfaces, you should use a higher level of abstraction when developing code for OS X. When you are trying to determine if a piece of code should be a KEXT, the default answer is generally no . Even if your code was a system extension in Mac OS 9, that does not necessarily mean that it should be a kernel extension in OS X. There are only a few good reasons for a developer to write a kernel extension: ● Your code needsto take a primary interrupt—that is,something in the (built-in) hardware needsto interrupt the CPU and execute a handler. ● The primary client of your code is inside the kernel—for example, a block device whose primary client is a file system. ● Your code needs to access kernel interfaces that are not exported to user space. ● Your code has other special requirements that cannot be satisfied in a user space application. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 150 Kernel Extension OverviewIf your code does not meet any of the above criteria (and possibly even if it does), you should consider developing it as a library or a user-level daemon, or using one of the user-level plug-in architectures (such as QuickTime components or the Core Graphics framework) instead of writing a kernel extension. If you are writing device drivers or code to support a new volume format or networking protocol, however, KEXTs may be the only feasible solution. Fortunately, while KEXTs may be more difficult to write than user-space code, several tools and procedures are available to enhance the development and debugging process. See “Debugging Your KEXT” (page 153) for more information. This chapter provides a conceptual overview of KEXTs and how to create them. If you are interested in building a simple KEXT, see the Apple tutorials listed in the bibliography. These provide step-by-step instructions for creating a simple, generic KEXT or a basic I/O Kit driver. Implementation of a Kernel Extension (KEXT) Kernel extensions are implemented as bundles, folders that the Finder treats as single files. See the chapter about bundles in Mac Technology Overview for a discussion of bundles.The KEXT bundle can contain the following: ● Information property list—a text file that describes the contents, settings, and requirements of the KEXT. This file is required. A KEXT bundle need contain nothing more than this file, although most KEXTs contain one or more kernel modules as well. See the chapter about software configuration in Mac Technology Overview for further information about property lists. ● KEXT binary—a file in Mach-O format, containing the actual binary code used by the KEXT. A KEXT binary (also known as a kernel module or KMOD) represents the minimum unit of code that can be loaded into the kernel. A KEXT usually contains one KEXT binary. If no KEXT binaries are included, the information property list file must contain a reference to another KEXT and change its default settings. ● Resources—for example, icons or localization dictionaries. Resources are optional; they may be useful for a KEXT that needs to display a dialog or menu. At present, no resources are explicitly defined for use with KEXTs. ● KEXT bundles—a kext can contain other KEXTs. This can be used for plug-ins that augment features of a KEXT. Kernel Extension Dependencies Any KEXT can declare that it is dependent upon any other KEXT. The developer lists these dependencies in the OSBundleLibraries dictionary in the module’s property list file. Kernel Extension Overview Implementation of a Kernel Extension (KEXT) 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 151Before a KEXT isloaded, all of itsrequirements are checked. Those required extensions(and their requirements) are loaded first, iterating back through the lists until there are no more required extensions to load. Only after all requirements are met, is the requested KEXT loaded as well. For example, device drivers (a type of KEXT) are dependent upon (require) certain families (another type of KEXT). When a driver isloaded, itsrequired families are also loaded to provide necessary, common functionality. To ensure that all requirements are met, each device drivershould list all of itsrequirements(families and other drivers) in its property list. See the chapter “I/O Kit Overview” (page 94), for an explanation of drivers and families. It is important to list all dependencies for each KEXT. If your KEXT fails to do so, your KEXT may not load due to unrecognized symbols, thusrendering the KEXT useless. Dependenciesin KEXTs can be considered analogous to required header files or librariesin code development; in fact, the Kernel Extension Manager usesthe standard linker to resolve KEXT requirements. Building and Testing Your Extension After creating the necessary property list and C or C++ source files, you use Project Builder to build your KEXT. Any errors in the source code are brought to your attention during the build and you are given the chance to edit your source files and try again. To test your KEXT, however, you need to leave Project Builder and work in the Terminal application (or in console mode). In console mode, all system messages are written directly to your screen, as well as to a log file (/var/log/system.log). If you work in the Terminal application, you must view system messages in the log file or in the Console application.You also need to log in to the root account (or use the su or sudo command), since only the root account can load kernel extensions. When testing your KEXT, you can load and unload it manually, as well as check the load status. You can use the kextload command to load any KEXT. A manual page for kextload is included in OS X. (On OS X prior to 10.2, you must use the kmodload command instead.) Note that this command is useful only when developing a KEXT. Eventually, after it has been tested and debugged, you install your KEXT in one of the standard places (see “Installed KEXTs” (page 154) for details). Then, it will be loaded and unloaded automatically at system startup and shutdown or whenever it is needed (such as when a new device is detected). Kernel Extension Overview Building and Testing Your Extension 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 152Debugging Your KEXT KEXT debugging can be complicated. Before you can debug a KEXT, you must first enable kernel debugging, as OS X is not normally configured to permit debugging the kernel. Only the root account can enable kernel debugging, and you need to reboot OS X for the changes to take effect. (You can use sudo to gain root privileges if you don’t want to enable a root password.) Kernel debugging is performed using two OS X computers, called the development or debug host and the debug target. These computers must be connected over a reliable network connection on the same subnet (or within a single local network). Specifically, there must not be any intervening IP routers or other devices that could make hardware-based Ethernet addressing impossible. The KEXT is registered (and loaded and run) on the target. The debugger is launched and run on the debug host. You can also rebuild your KEXT on the debug host, after you fix any errors you find. Debugging must be performed in this fashion because you must temporarily halt the kernel on the target in order to use the debugger. When you halt the kernel, all other processes on that computer stop. However, a debugger running remotely can continue to run and can continue to examine (or modify) the kernel on the target. Note that bugs in KEXTs may cause the target kernel to freeze or panic. If this happens, you may not be able to continue debugging, even over a remote connection; you have to reboot the target and start over, setting a breakpoint just before the code where the KEXT crashed and working very carefully up to the crash point. Developers generally debug KEXTs using gdb, a source-level debugger with a command-line interface. You will need to work in the Terminal application to run gdb. For detailed information about using gdb, see the documentation included with OS X. You can also use the help command from within gdb. Some features of gdb are unavailable when debugging KEXTs because of implementation limitations. For example: ● You can’t use gdb to call a function or method in a KEXT. ● You should not use gdb to debug interrupt routines. The former is largely a barrier introduced by the C++ language. The latter may work in some cases but is not recommended due to the potential for gdb to interrupt something upon which kdp (the kernel shim used by gdb) depends in order to function properly. Use care that you do not halt the kernel for too long when you are debugging (for example, when you set breakpoints). In a short time, internal inconsistencies can appear that cause the target kernel to panic or freeze, forcing you to reboot the target. Kernel Extension Overview Debugging Your KEXT 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 153Additional information about debugging can be found in “When Things Go Wrong: Debugging the Kernel” (page 161). Installed KEXTs The Kernel Extension Manager (KEXT Manager) is responsible for loading and unloading all installed KEXTs (commands such as kextload are used only during development). Installed KEXTs are dynamically added to the running OS X kernel as part of the kernel’s address space. An installed and enabled KEXT is invoked as needed. Important: Note that KEXTs are only wrappers(bundles) around a property list, KEXT binaries(or references to other KEXTs), and optional resources. The KEXT describes what is to be loaded; it is the KEXT binaries that are actually loaded. KEXTs are usually installed in the folder /System/Libraries/Extensions. The Kernel Extension Manager (in the form of a daemon, kextd), always checks here. KEXTs can also be installed in ROM or inside an application bundle. Installing KEXTs in an application bundle allows an application to register those KEXTs without the need to install them permanently elsewhere within the system hierarchy. This may be more convenient and allows the KEXT to be associated with a specific, running application. When it starts, the application can register the KEXT and, if desired, unregister it on exit. For example, a network packet sniffer application might employ a Network Kernel Extension (NKE). A tape backup application would require that a tape driver be loaded during the duration of the backup process. When the application exits, the kernel extension is no longer needed and can be unloaded. Note that, although the application is responsible for registering the KEXT, this is no guarantee that the corresponding KEXTs are actually ever loaded. It is still up to a kernel component, such as the I/O Kit, to determine a need, such as matching a piece of hardware to a desired driver, thus causing the appropriate KEXTs (and their dependencies) to be loaded. Kernel Extension Overview Installed KEXTs 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 154This chapter is not about building kernel extensions (KEXTs). There are a number of good KEXT tutorials on Apple’s developer documentation site (http://developer.apple.com/documentation). This chapter is about adding new in-kernel modules(optional parts of the kernel), building kernels, and debugging kernel and kernel extension builds. The discussion is divided into three sections. The first, “Adding New Files or Modules” (page 155), describes how to add new functionality into the kernel itself. You should only add files into the kernel when the use of a KEXT is not possible (for example, when adding certain low-level motherboard hardware support). The second section, “Building Your First Kernel” (page 158), describes how to build a kernel, including how to build a kernel with debugger support, how to add new options, and how to obtain sources that are of similar vintage to those in a particular version of OS X or Darwin. The third section, “When Things Go Wrong: Debugging the Kernel” (page 161), tells how to debug a kernel or kernel module using ddb and gdb. This is a must-read for anyone doing kernel development. Adding New Files or Modules In this context, the term module is used loosely to refer to a collection of related files in the kernel that are controlled by a single config option at compile time. It does not refer to loadable modules (KEXTs). This section describes how to add additional files that will be compiled into the kernel, including how to add a new config option for an additional module. Modifying the Configuration Files The details of adding a new file or module into the kernel differ according to what portion of the kernel contains the file. If you are adding a new file or module into the Mach portion of the kernel, you need to list it in various filesin xnu/osfmk/conf. For the BSD portion of the kernel, you should list it in variousfilesin xnu/bsd/conf. In either case, the procedure is basically the same, just in a different directory. This section is divided into two subsections. The first describes adding the module itself and the second describes enabling the module. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 155 Building and Debugging KernelsAdding the Files or Modules In the appropriate conf directory, you need to add your files or modules into various files. The files MASTER, MASTER.ppc, and MASTER.i386 contain the list of configuration options that should be built into the kernel for all architectures, PowerPC, and i386, respectively. These are supplemented by files, files.ppc, and files.i386, which contain associations between compile options and the files that are related to them for their respective architectures. The format for these two files is relatively straightforward. If you are adding a new module, you should first choose a name for that module. For example, if your module is called mach_foo, you should then add a new option line near the top of files that is whitespace (space or tab) delimited and looks like this: OPTIONS/mach_foo optional mach_foo The first part defines the name of the module as it will be used in #if statements in the code. (See “Modifying the Source Code Files” (page 157) for more information.) The second part is alwaysthe word optional. The third part tells the name of the option as used to turn it on or off in a MASTER file. Any line with mach_foo in the last field will be enabled only if there is an appropriate line in a MASTER file. Then, later in the file, you add osfmk/foo/foo_main.c optional mach_foo osfmk/foo/foo_bar.c optional mach_foo and so on, for each new file associated with that module. This also applies if you are adding a file to an existing module. If you are adding a file that is not associated with any module at all, you add a line that looks like the following to specify that this file should always be included: osfmk/crud/mandatory_file.c standard If you are not adding any modules, then you’re done. Otherwise, you also need to enable your option in one of the MASTER files. Enabling Module Options To enable a module option (as described in the files files), you must add an entry for that option into one of the MASTER files. If your code is not a BSD pseudo-device, you should add something like the following: options MACH_FOO Building and Debugging Kernels Adding New Files or Modules 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 156Otherwise, you should add something like this: pseudo-device mach_foo In the case of a pseudo-device (for example, /dev/random), you can also add a number. When your code checks to see if it should be included, it can also check that number and allocate resources for more than one pseudo-device. The meaning of multiple pseudo-devicesis device-dependent. An example of thisis ppp, which allocates resources for two simultaneous PPP connections. Thus, in the MASTER.ppc file, it has the line: pseudo-device ppp 2 Modifying the Source Code Files In the OS X kernel, all source code files are automatically compiled. It is the responsibility of the C file itself to determine whether its contents need to be included in the build or not. In the example above, you created a module called mach_foo. Assume that you want this file to compile only on PowerPC-based computers. In that case, you should have included the option only in MASTER.ppc and not in MASTER.i386. However, by default, merely specifying the file foo_main.c in files causes it to be compiled, regardless of compile options specified. To make the code compile only when the option mach_foo is included in the configuration, you should begin each C source file with the lines #include #if (MACH_FOO > 0) and end it with #endif /* MACH_FOO */ If mach_foo is a pseudo-device and you need to check the number of mach_foo pseudo-devices included, you can do further tests of the value of MACH_FOO. Note that the file is not something you create. It is created by the makefiles themselves. You must run make exporthdrs before make all to generate these files. Building and Debugging Kernels Adding New Files or Modules 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 157Building Your First Kernel Before you can build a kernel, you must first obtain source code. Source code for the OS X kernel can be found in the Darwin xnu project on http://www.opensource.apple.com. To find out your current kernel version, use the command uname -a. If you run into trouble, search the archives of the darwin-kernel and darwin-development mailing lists for information. If that doesn’t help, ask for assistance on either list. The list archives and subscription information can be found at http://www.lists.apple.com. Note: Before you begin, make sure you extract the sources in a directory whose path does not contain any “special” characters (non-alphanumeric characters other than dash and underscore), as having such characters in the path leading up to the build directory can cause compiling to fail. Also, make sure that /usr/local/bin is in your PATH environment variable as follows: If you are using a csh derivative such as tcsh, you should add set path = (/usr/local/bin $path) to your .tcshrc file If you are using a Bourne shell derivative, you should add export PATH=/usr/local/bin:$PATH to your .bashrc file. Important: Once you have obtained and extracted the sources, before you begin compiling kernelsupport tools, you should configure your system to build using gcc 3.3. The OS X v10.4 kernel will not build using gcc 4.0. To do this, type: sudo gcc_select 3.3 Important: Before building anything, you should make sure you are running the latest version of OS X with the latest developer tools. The xnu compile process may reference various external headers from /System/Library/Frameworks. These headers are only installed as part of a developer toolsinstallation, not as part of the normal OS X install process. Next, you will need to compile several support tools. Get the bootstrap_cmds, Libstreams, kext_tools, IOKitUser, and cctools packagesfrom http://www.opensource.apple.com. Extract the filesfrom these .tar packages, then do the following: sudo mkdir -p /usr/local/bin sudo mkdir -p /usr/local/lib cd bootstrap_cmds-version/relpath.tproj make Building and Debugging Kernels Building Your First Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 158sudo make install cd ../../Libstreams-version make sudo make install cd ../cctools-version sudo cp /usr/include/ar.h \ /System/Library/Frameworks/Kernel.framework/Headers In the cctools package, modify the Makefile, and change the COMMON_SUBDIRS line (including the continuation line after it) to read: COMMON_SUBDIRS = libstuff libmacho misc Finally, issue the following commands: make RC_OS=macos sudo cp misc/seg_hack.NEW /usr/local/bin/seg_hack cd ld make RC_OS=macos kld_build sudo cp static_kld/libkld.a /usr/local/lib sudo ranlib /usr/local/lib/libkld.a Now you’re done with the cctools project. One final step remains: compiling kextsymboltool. To do this, extract the kext_tools tarball, then do the following: sudo mkdir -p /System/Library/Frameworks/IOKit.framework/Versions/A/PrivateHeaders/kext cd /System/Library/Frameworks/IOKit.framework/ sudo ln -s Versions/A/PrivateHeaders PrivateHeaders sudo cp PATH_TO_IOKITUSER/IOKitUser-version/kext.subproj/*.h PrivateHeaders/kext cd PATH_TO_KEXT_TOOLS/kext_tools-version gcc kextsymboltool.c -o kextsymboltool sudo cp kextsymboltool /usr/local/bin Building and Debugging Kernels Building Your First Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 159Warning: If you do not use a version of kextsymboltool that is at least as current as your kernel, you will get serious compile failures. If you see the error message “exported name not in import list”, there’s a good chance you aren’t using a current kextsymboltool. Congratulations. You now have all the necessary tools, libraries, and header files to build a kernel. The next step is to compile the kernel itself. First, change directories into the xnu directory. Next, you need to set a few environment variables appropriately. For your convenience, the kernel sources contain shell scripts to do this for you. If you are using sh, bash, zsh, or some other Bourne-compatible shell, issue the following command: source SETUP/setup.sh If you are using csh, tcsh, or a similar shell, use the following command: source SETUP/setup.csh Then, you should be able to type make exporthdrs make all and get a working kernel in BUILD/obj/RELEASE_PPC/mach_kernel (assuming you are building a RELEASE kernel for PowerPC, of course). If things don’t work, the darwin-kernel mailing list a good place to get help. Building an Alternate Kernel Configuration When building a kernel, you may want to build a configuration other than the RELEASE configuration (the default shipping configuration). Additional configurations are RELEASE_TRACE, DEBUG, DEBUG_TRACE, and PROFILE. These configurations add various additional options (except PROFILE, which is reserved for future expansion, and currently maps onto RELEASE). Building and Debugging Kernels Building an Alternate Kernel Configuration 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 160The most useful and interesting configurations are RELEASE and DEBUG. The release configuration should be the same as a stock Apple-released kernel, so this is interesting only if you are building source that differs from that which was used to build the kernel you are already running. Compiling a kernel without specifying a configuration results in the RELEASE configuration being built. The DEBUG configuration enables ddb, the in-kernel serial debugger. The ddb debugger is helpful to debug panics that occur early in boot or within certain parts of the Ethernet driver. It is also useful for debugging low-level interrupt handler routines that cannot be debugged by using the more traditional gdb. To compile an alternate kernel configuration, you should follow the same basic procedure as outlined previously, changing the final make statement slightly. For example, to build the DEBUG configuration, instead of typing make all you type make KERNEL_CONFIGS=DEBUG all and wait. To turn on additional compile options, you must modify one of the MASTER files. For information on modifying these files, see the section “Enabling Module Options” (page 156). When Things Go Wrong: Debugging the Kernel No matter how careful your programming habits, sometimes things don’t work right the first time. Kernel panics are simply a fact of life during development of kernel extensions or other in-kernel code. There are a number of ways to track down problems in kernel code. In many cases, you can find the problem through careful use of printf or IOLog statements. Some people swear by this method, and indeed, given sufficient time and effort, any bug can be found and fixed without using a debugger. Of course, the key words in that statement are “given sufficient time and effort.” For the rest of us, there are debuggers: gdb and ddb. Setting Debug Flags in Open Firmware With the exception of kernel panics or calls to PE_enter_debugger, it is not possible to do remote kernel debugging without setting debug flags in Open Firmware. These flags are relevant to both gdb and ddb debugging and are important enough to warrant their own section. Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 161To set these flags, you can either use the nvram program (from the OS X command line) or access your computer’s Open Firmware. You can access Open Firmware this by holding down Command-Option-O-F at boot time. For most computers, the default is for Open Firmware to present a command–line prompt on your monitor and accept input from your keyboard. For some older computers you must use a serial line at 38400, 8N1. (Technically, such computers are not supported by OS X, but some are usable under Darwin, and thus they are mentioned here for completeness.) From an Open Firmware prompt, you can set the flags with the setenv command. From the OS X command line, you would use the nvram command. Note that when modifying these flags you should always look at the old value for the appropriate Open Firmware variables and add the debug flags. For example, if you want to set the debug flagsto 0x4, you use one of the following commands. For computers with recent versions of Open Firmware, you would type printenv boot-args setenv boot-args original_contents debug=0x4 from Open Firmware or nvram boot-args nvram boot-args="original_contents debug=0x4" from the command line (as root). For older firmware versions, the interesting variable is boot-command. Thus, you might do something like printenv boot-command setenv boot-command 0 bootr debug=0x4 from Open Firmware or nvram boot-command nvram boot-command="0 bootr debug=0x4" from the command line (as root). Of course, the more important issue is what value to choose for the debug flags. Table 20-1 (page 163) lists the debugging flags that are supported in OS X. Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 162Table 20-1 Debugging flags Symbolic name Flag Meaning DB_HALT 0x01 Halt at boot-time and wait for debugger attach (gdb). DB_PRT 0x02 Send kernel debugging printf output to console. Drop into debugger on NMI (Command–Power, Command-Option-Control-Shift-Escape, or interrupt switch). DB_NMI 0x04 DB_KPRT 0x08 Send kernel debugging kprintf output to serial port. DB_KDB 0x10 Make ddb (kdb) the default debugger (requires a custom kernel). DB_SLOG 0x20 Output certain diagnostic info to the system log. Allow debugger to ARP and route (allows debugging across routers and removes the need for a permanent ARP entry, but is a potential security hole)—not available in all kernels. DB_ARP 0x40 DB_KDP_BP_DIS 0x80 Support old versions of gdb on newer systems. DB_LOG_PI_SCRN 0x100 Disable graphical panic dialog. The option DB_KDP_BP_DIS is not available on all systems, and should not be important if your target and host systems are running the same or similar versions of OS X with matching developer tools. The last option is only available in Mac OS 10.2 and later. Avoiding Watchdog Timer Problems Macintosh computers have various watchdog timers designed to protect the system from certain types of failures. There are two primary watchdog timersin common use: the power management watchdog timer (not present on all systems) and the system crash watchdog timer. Both watchdogs are part of the power management hardware. The first of these, the power management watchdog timer, is designed to restore the system to a known safe state in the event of unexpected communication loss between the power management hardware and the CPU. Thistimer is only present in G4 and earlier desktops and laptops and in early G5 desktops. More specifically, it is present only in machines containing a PMU (Power Management Unit) chip. Under normal circumstances, when communication with the PMU chip is lost, the PMU driver will attempt to get back in sync with the PMU chip. With the possible exception of a momentary loss of keyboard and mouse control, you probably won't notice that anything has happened (and you should never even experience such a stall unless you are writing a device driver that disables interrupts for an extended period of time). Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 163The problem occurs when the disruption in communication is caused by entering the debugger while the PMU chip is in one of these "unsafe" states. If the chip is left in one of these "unsafe" states for too long, it will shut the computer down to prevent overheating or other problems. This problem can be significantly reduced by operating the PMU chip in polled mode. This prevents the watchdog timer from activating. You should only use this option when debugging, however, as it diminishes performance and a crashed system could overheat. To disable this watchdog timer, add the argument pmuflags=1 to the kernel's boot arguments. See “Setting Debug Flags in Open Firmware” (page 161) for information about how to add a boot argument. The second type of watchdog timer is the system crash watchdog timer. This is normally only enabled in OS X Server. If your target machine is running OS X Server, your system will automatically reboot within seconds after a crash to maximize server uptime. You can disable this automatic reboot on crash feature in the server administration tool. Choosing a Debugger There are two basic debugging environments supported by OS X: ddb and gdb. ddb is a built-in debugger that works over a serial line. By contrast, gdb is supported using a debugging shim built into the kernel, which allows a remote computer on the same physical network to attach after a panic (or sooner if you pass certain options to the kernel). For problems involving network extensions or low-level operating system bringups, ddb is the only way to do debugging. For other bugs, gdb is generally easier to use. For completeness, this chapter describes how to use both ddb and gdb to do basic debugging. Since gdb itself is well documented and is commonly used for application programming, this chapter assumes at least a passing knowledge of the basics of using gdb and focuses on the areas where remote (kernel) gdb differs. Note: Only systems with serial hardware support ddb. Thus, it is only possible to use ddb on PowerMac G4 and older systems. Using gdb for Kernel Debugging gdb, short for the GNU Debugger, is a piece of software commonly used for debugging software on UNIX and Linux systems. This section assumes that you have used gdb before, and does not attempt to explain basic usage. In standard OS X builds (and in your builds unless you compile with ddb support), gdb support is built into the system but is turned off except in the case of a kernel panic. Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 164Of course, many software failures in the kernel do not result in a kernel panic but still cause aberrant behavior. For these reasons, you can pass additional flags to the kernel to allow you to attach to a remote computer early in boot or after a nonmaskable interrupt (NMI), or you can programmatically drop into the debugger in your code. You can cause the test computer (the debug target) to drop into the debugger in the following ways: ● debug on panic ● debug on NMI ● debug on boot ● programmatically drop into the default debugger The function PE_enter_debugger can be called from anywhere in the kernel, although if gdb is your default debugger, a crash will result if the network hardware is not initialized or if gdb cannot be used in that particular context. This call is described in the header pexpert/pexpert.h. After you have decided what method to use for dropping into the debugger on the target, you must configure your debug host (the computer that will actually be running gdb). Your debug hostshould be running a version of OS X that is comparable to the version running on your target host. However, it should not be running a customized kernel, since a debug host crash would be problematic, to say the least. Note: It is possible to use a non-OS X system as your debug host. This is not a trivial exercise, however, and a description of building a cross-gdb is beyond the scope of this document. When using gdb, the best results can be obtained when the source code for the customized kernel is present on your debug host. This not only makes debugging easier by allowing you to see the lines of code when you stop execution, it also makes it easier to modify those lines of code. Thus, the ideal situation is for your debug host to also be your build computer. This is not required, but it makes things easier. If you are debugging a kernel extension, it generally suffices to have the source for the kernel extension itself on your debug host. However, if you need to see kernel-specific structures, having the kernel sources on your debug host may also be helpful. Once you have built a kernel using your debug host, you must then copy it to your target computer and reboot the target computer. At this point, if you are doing panic-only debugging, you should trigger the panic. Otherwise, you should tell your target computer to drop into the debugger by issuing an NMI (or by merely booting, in the case of debug=0x1). Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 165Next, unless your kernelsupports ARP while debugging (and unless you enabled it with the appropriate debug flag), you need to add a permanent ARP entry for the target. It will be unable to answer ARP requests while waiting for the debugger. This ensures that your connection won’t suddenly disappear. The following example assumes that your target is target.foo.com with an IP number of 10.0.0.69: $ ping -c 1 target_host_name ping results: .... $ arp -an target.foo.com (10.0.0.69): 00:a0:13:12:65:31 $ sudo arp -s target.foo.com 00:a0:13:12:65:31 $ arp -an target.foo.com (10.0.0.69) at00:a0:13:12:65:31 permanent Now, you can begin debugging by doing the following: gdb /path/to/mach_kernel source /path/to/xnu/osfmk/.gdbinit p proc0 source /path/to/xnu/osfmk/.gdbinit target remote-kdp attach 10.0.0.69 Note that the mach kernel passed as an argument to gdb should be the symbol–laden kernel file located in BUILD/obj/DEBUG_PPC/mach_kernel.sys (for debug kernel builds, RELEASE_PPC for non-debug builds), not the bootable kernel that you copied onto the debug target. Otherwise most of the gdb macros will fail. The correct kernel should be several times as large as a normal kernel. You must do the p proc0 command and source the .gdbinit file (from the appropriate kernel sources) twice to work around a bug in gdb. Of course, if you do not need any of the macros in .gdbinit, you can skip those two instructions. The macros are mostly of interest to people debugging aspects of Mach, though they also provide ways of obtaining information about currently loaded KEXTs. Warning: It may not be possible to detach in a way that the target computer’s kernel continues to run. If you detach, the target hangs until you reattach. It is not always possible to reattach, though the situation is improving in this area. Do not detach from the remote kernel! Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 166If you are debugging a kernel module, you need to do some additional work to get debugging symbol information about the module. First, you need to know the load address for the module. You can get this information by running kextstat (kmodstat on systems running OS X v10.1 or earlier) as root on the target. If you are already in the debugger, then assuming the target did not panic, you should be able to use the continue function in gdb to revive the target, get this information, then trigger another NMI to drop back into the debugger. If the target is no longer functional, and if you have a fully symbol–laden kernel file on your debug host that matches the kernel on your debug target, you can use the showallkmods macro to obtain this information. Obtaining a fully symbol–laden kernel generally requires compiling the kernel yourself. Once you have the load address of the module in question, you need to create a symbol file for the module. You do this in different ways on different versions of OS X. For versions 10.1 and earlier, you use the kmodsyms program to create a symbol file for the module. If your KEXT is called mykext and it is loaded at address 0xf7a4000, for example, you change directories to mykext.kext/Contents/MacOS and type: kmodsyms -k path/to/mach_kernel -o mykext.sym mykext@0xf7a4000 Be sure to specify the correct path for the mach kernel that is running on your target (assuming it is not the same as the kernel running on your debug host). For versions after 10.1, you have two options. If your KEXT does not crash the computer when it loads, you can ask kextload to generate the symbols at load time by passing it the following options: kextload -s symboldir mykext.kext It will then write the symbols for your kernel extension and its dependencies into files within the directory you specified. Of course, this only works if your target doesn’t crash at or shortly after load time. Alternately, if you are debugging an existing panic, or if your KEXT can’t be loaded without causing a panic, you can generate the debugging symbols on your debug host. You do this by typing: kextload -n -s symboldir mykext.kext If will then prompt you for the load address of the kernel extension and the addresses of all its dependencies. As mentioned previously, you can find the addresses with kextstat (or kmodstat) or by typing showallkmods inside gdb. Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 167You should now have a file or files containing symbolic information that gdb can use to determine address–to–name mappings within the KEXT. To add the symbols from that KEXT, within gdb on your debug host, type the command add-symbol-file mykext.sym for each symbol file. You should now be able to see a human-readable representation of the addresses of functions, variables, and so on. Special gdb I/O Addressing Issues As described in “Address Spaces” (page 70), some Macintosh hardware has a third addressing mode called I/O addressing which differs from both physical and virtual addressing modes. Most developers will not need to know about these modes in any detail. Where some developers may run into problems is debugging PCI device drivers and attempting to access device memory/registers. To allow I/O-mapped memory dumping, do the following: set kdp_read_io=1 To dump in physical mode, do the following: set kdp_trans_off=1 For example: (gdb) x/x 0xf8022034 0xf8022034: Cannot access memory at address 0xf8022034 (gdb) set kdp_trans_off=1 (gdb) x/x 0xf8022034 0xf8022034: Cannot access memory at address 0xf8022034 (gdb) set kdp_read_io=1 (gdb) x/x 0xf8022034 0xf8022034: 0x00000020 (gdb) Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 168If you experience problems accessing I/O addresses that are not corrected by this procedure, please contact Apple Developer Technical Support for additional assistance. Using ddb for Kernel Debugging When doing typical debugging, gdb is probably the best solution. However, there are times when gdb cannot be used or where gdb can easily run into problems. Some of these include ● drivers for built-in Ethernet hardware ● interrupt handlers (the hardware variety, not handler threads) ● early bootstrap before the network hardware is initialized When gdb is not practical (or if you’re curious), there is a second debug mechanism that can be compiled into OS X. This mechanism is called ddb, and is similar to the kdb debugger in most BSD UNIX systems. It is not quite as easy to use as gdb, mainly because of the hardware needed to use it. Unlike gdb (which uses Ethernet for communication with a kernel stub), ddb is built into the kernel itself, and interacts directly with the user over a serial line. Also unlike gdb, using ddb requires building a custom kernel using the DEBUG configuration. For more information on building this kernel, see “Building Your First Kernel” (page 158). Note: ddb requires an actual built-in hardware serial line on the debug target. Neither PCI nor USB serial adapters will work. In order to work reliably for interrupt-level debugging, ddb controls the serial ports directly with a polled-mode driver without the use of the I/O Kit. If your debug target does not have a factory serial port, third-party adapter boards may be available that replace your internal modem with a serial port. Since these devices use the built-in serial controller, they should work for ddb. It is not necessary to install OS X drivers for these devices if you are using them only to support ddb debugging. The use of these serial port adapter cards is not an officially supported configuration, and not all computers support the third-party adapter boards needed for ddb support. Consult the appropriate adapter board vendor for compatibility information. If your target computer has two serial ports, ddb uses the modem port (SCC port 0). However, if your target has only one serial port, that port is probably attached to port 1 of the SCC cell, which means that you have to change the default port if you want to use ddb. To use this port (SCC port 1), change the line: const int console_unit=0; Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 169in osfmk/ppc/serial_console.c to read: const int console_unit=1; and recompile the kernel. Once you have a kernel with ddb support, it isrelatively easy to use. First, you need to set up a terminal emulator program on your debug host. If your debug host is running Mac OS 9, you might use ZTerm, for example. For OS X computers, or for computers running Linux or UNIX, minicom provides a good environment. Setting up these programs is beyond the scope of this document. Important: Serial port settings for communicating with ddb must be 57600 8N1. Hardware handshaking may be on, but is not necessary. Note: For targets whose Open Firmware uses the serial ports, remember that the baud rate for communicating with Open Firmware is 38400 and that hardware handshaking must be off. Once you boot a kernel with ddb support, a panic will allow you to drop into the debugger, as will a call to PE_enter_debugger. If the DB_KDB flag is not set, you will have to press the D key on the keyboard to use ddb. Alternately, if both DB_KDB and DB_NMI are set, you should be able to drop into ddb by generating a nonmaskable interrupt (NMI). See “Setting Debug Flags in Open Firmware” (page 161) for more information on debug flags. To generate a nonmaskable interrupt, hold down the command, option, control, and shift keys and hit escape (OS X v10.4 and newer), hold down the command key while pressing the power key on your keyboard (on hardware with a power key), or press the interrupt button on your target computer. At this point, the system should hang, and you should see ddb output on the serial terminal. If you do not, check your configuration and verify that you have specified the correct serial port on both computers. Commands and Syntax of ddb The ddb debugger is much more gdb-like than previous versions, but it still has a syntax that is very much its own (shared only with other ddb and kdb debuggers). Because ddb is substantially different from what most developers are used to using, this section outlines the basic commands and syntax. The commands in ddb are generally in this form: command[/switch] address[,count] Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 170The switches can be one of those shown in Table 20-2 (page 171). Table 20-2 Switch options in ddb Switch Description /A Print the location with line number if possible /I Display as instruction with possible alternate machine-dependent format /a Print the location being displayed /b Display or process by bytes Display low 8 bits as a character (nonprinting characters as octal) or count instructions while executing (depends on instruction) /c /d Display as signed decimal /h Display or process by half word (16 bits) /i Display as an instruction /l Display or process by long word (32 bits) /m Display as unsigned hex with character dump for each line /o Display in unsigned octal /p Print cumulative instruction count and call tree depth at each call or return statement /r Display in current radix, signed /s Display the null-terminated string at address (nonprinting as octal). Display in unsigned decimal or set breakpoint at a user space address (depending on command). /u /x Display in unsigned hex /z Display in signed hex The ddb debugger has a rich command set that has grown over its lifetime. Its command set is similar to that of ddb and kdb on other BSD systems, and their manual pages provide a fairly good reference for the various commands. The command set for ddb includes the following commands: Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 171break[/u] addr Set a breakpoint at the address specified by addr. Execution will stop when the breakpoint is reached. The /u switch means to set a breakpoint in user space. c or continue[/c] Continue execution after reaching a breakpoint. The /c switch meansto count instructions while executing. call Call a function. cond Set condition breakpoints. This command is not supported on PowerPC. cpu cpunum Causes ddb to switch to run on a different CPU. d or delete [addr|#] Delete a breakpoint. This takes a single argument that can be either an address or a breakpoint number. dk Equivalent to running kextstat while the target computer is running. This lists loaded KEXTs, their load addresses, and various related information. dl vaddr Dumps a range of memory starting from the address given. The parameter vaddr is a kernel virtual address. If vaddr is not specified, the last accessed address is used. See also dr, dv. dm Displays mapping information for the last address accessed. dmacro name Delete the macro called name. See macro. dp Displays the currently active page table. dr addr Dumps a range of memory starting from the address given. The parameter address is a physical address. If addr is not specified, the last accessed address is used. See also dl, dv. ds Dumps save areas of all Mach tasks. dv [addr [vsid]] Dumps a range of memory starting from the address given. The parameter addr is a virtual address in the address space indicated by vsid. If addr is not specified, the last accessed address is used. Similarly, if vsid is not specified, the last vsid is used. See also dl, dr. Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 172dwatch addr Delete a watchpoint. See watch. dx Displays CPU registers. examine See print. gdb Switches to gdb mode, allowing gdb to attach to the computer. lt On PowerPC only: Dumps the PowerPC exception trace table. macro name command [ ; command .. ] Create a macro called name that executesthe listed commands. You can show a macro with the command show macro name or delete it with dmacro name. match[/p] Stop at the matching return instruction. If the /p switch is not specified, summary information is printed only at the final return. print[/AIabcdhilmorsuxz] addr1 [addr2 ...] Print the values at the addresses given in the format specified by the switch. If no switch is given, the last used switch is assumed. Synonymous with examine and x. Note that some of the listed switches may work for examine and not for print. reboot Reboots the computer. Immediately. Without doing any file-system unmounts or other cleanup. Do not do this except after a panic. s or step Single step through instructions. search[/bhl] addr value [mask[,count]] Search memory for value starting at addr. If the value is not found, this command can wreak havoc. This command may take other formatting values in addition to those listed. set $name [=] expr Sets the value of the variable or register named by name to the value indicated by expr. show Display system data. For a list of information that can be shown, type the show command by itself. Some additional options are available for certain options, particularly show all. For those suboptions, type show all by itself. Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 173trace[/u] Prints a stack backtrace. If the /u flag is specified, the stack trace extends to user space if supported by architecture-dependent code. until[/p] Stop at the next call or return. w or write[/bhl] addr expr1 [expr2 ... ] Writes the value of expr1 to the memory location stored at addr in increments of a byte, half word, or long word. If additional expressions are specified, they are written to consecutive bytes, half words, or long words. watch addr[,size] Sets a watchpoint on a particular address. Execution stops when the value stored at that address is modified. Watch points are not supported on PowerPC. Warning: Watching addresses in wired kernel memory may cause unrecoverable errors on i386. x Short for examine. See print. xb Examine backward. Execute the last examine command, but use the address previous to the last one used (jumping backward by increments of the last width displayed). xf Examine forward. Execute the last examine command, but use the address following the last one used (jumping by increments of the last width displayed). The ddb debugger should seem relatively familiar to users of gdb, and its syntax was changed radically from its predecessor, kdb, to be more gdb-like. However, it is still sufficiently different that you should take some time to familiarize yourself with its use before attempting to debug something with it. It is far easier to use ddb on a system whose memory hasn’t been scribbled upon by an errant DMA request, for example. Building and Debugging Kernels When Things Go Wrong: Debugging the Kernel 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 174This bibliography contains related material that may be of interest. The editions listed are the editions that were current when this list was compiled, but newer versions may be available. Apple OS X Publications The following Apple publications have information that could be of interest to you if you are programming in the kernel: Hello Debugger: Debugging a Device Driver With GDB (tutorial). Hello I/O Kit: Creating a Device Driver With Xcode (tutorial) Hello Kernel: Creating a Kernel Extension With Xcode (tutorial). Accessing Hardware From Applications I/O Kit Fundamentals Network Kernel Extensions Programming Guide Network Kernel Extensions (legacy) Mac Technology Overview Porting UNIX/Linux Applications to OS X I/O Kit Device Driver Design Guidelines Packaging Your KEXT for Distribution and Installation(tutorial). General UNIX and Open Source Resources A Quarter Century of UNIX . Peter H. Salus. Addison-Wesley, 1994.ISBN 0-201-54777-5. Berkeley Software Distribution . CSRG, UC Berkeley. USENIX and O’Reilly, 1994.ISBN 1-56592-082-1. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 175 BibliographyTheCathedralandtheBazaar:MusingsonLinuxandOpenSourcebyanAccidentalRevolutionary . Eric S.Raymond. O’Reilly & Associates, 1999.ISBN 1-56592-724-9. The New Hacker’s Dictionary . 3rd. Ed., Eric S. Raymond. MIT Press, 1996. ISBN 0-262-68092-0. Open Sources: Voices from the Open Source Revolution . Edited by Chris DiBona, Sam Ockman & Mark Stone. O’Reilly & Associates, 1999. ISBN 1-56592-582-3. Proceedings of the First Conference on Freely Redistributable Software . Free Software Foundation. FSF, 1996. ISBN 1-882114-47-7. The UNIX Desk Reference: The hu.man Pages. Peter Dyson. Sybex, 1996. ISBN 0-7821-1658-2. The UNIX Programming Environment. Brian W. Kernighan, Rob Pike. Prentice Hall, 1984. ISBN 0-13-937681-X (paperback), ISBN 0-13-937699-2 (hardback). BSD and UNIX Internals Advanced Topics in UNIX: Processes, Files, and Systems. Ronald J. Leach. Wiley, 1996. ISBN 1-57176-159-4. The Complete FreeBSD. Greg Lehey, Walnut Creek CDROM Books, 1999. ISBN 1-57176-246-9. The Design and Implementation of the 4.4BSD Operating System. Marshall Kirk McKusick, et al. Addison-Wesley, 1996. ISBN 0-201-54979-4. The Design of the UNIX Operating System. Maurice J. Bach. Prentice Hall, 1986. ISBN 0-13-201799-7. Linux Kernel Internals 2nd edition . Michael Beck, et al. Addison-Wesley, 1997. ISBN 0-201-33143-8. Lions’ Commentary on UNIX 6th Edition with Source Code . John Lions. Peer-to-Peer, 1996. ISBN 1-57398-013-7. Panic!: UNIX System Crash Dump Analysis. Chris Drake, Kimberly Brown. Prentice Hall, 1995. ISBN 0-13-149386-8. UNIX Internals: The New Frontiers. Uresh Vahalia. Prentice-Hall, 1995. ISBN 0-13-101908-2. UNIX Systems for Modern Architectures: Symmetric Multiprocessing and Caching for Kernel Programmers. Curt Schimmel. Addison-Wesley, 1994. ISBN 0-201-63338-8. Optimizing PowerPC Code . Gary Kacmarcik. Addison-Wesley Publishing Company, 1995. ISBN 0-201-40839-2. BerkeleySoftwareArchitectureManual4.4BSDEdition .WilliamJoy,Robert Fabry, Samuel Leffler,M.KirkMcKusick, Michael Karels. Computer Systems Research Group, Computer Science Division, Department of Electrical Engineering and Computer Science, University of California, Berkeley. Bibliography BSD and UNIX Internals 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 176Mach CMU Computer Science: A 25th Anniversary Commemorative . Richard F. Rashid, Ed. ACM Press, 1991. ISBN 0-201-52899-1. Load Distribution, Implementation for the Mach Microkernel . Dejan S. Milojicic. Vieweg Verlag, 1994. ISBN 3-528-05424-7. Programming under Mach . Boykin, et al. Addison-Wesley, 1993. ISBN 0-201-52739-1. Mach Workshop Proceedings. USENIX Association. October, 1990. Mach Symposium Proceedings.USENIX Association. November, 1991. Mach III Symposium Proceedings. USENIX Association. April, 1993, ISBN 1-880446-49-9. Mach 3 Documentation Series. Open Group Research Institute (RI), now Silicomp: Final Draft Specifications OSF/1 1.3 Engineering Release . RI. May 1993. OSF Mach Final Draft Kernel Principles. RI. May, 1993. OSF Mach Final Draft Kernel Interfaces. RI. May, 1993. OSF Mach Final Draft Server Writer’s Guide . RI. May, 1993. OSF Mach Final Draft Server Library Interfaces, RI, May, 1993. Research Institute Microkernel Series. Open Group Research Institute (RI): Operating Systems Collected Papers. Volume I. RI. March, 1993. Operating Systems Collected Papers. Volume II. RI. October,1993. Operating Systems Collected Papers. Volume III. RI. April, 1994. Operating Systems Collected Papers. Volume IV. RI. October, 1995. Mach: A New Kernel Foundation for UNIX Development. Proceedings of the Summer 1986 USENIX Conference. Atlanta, GA., http://www.usenix.org. UNIX as an Application Program. Proceedings of the Summer 1990 USENIX Conference. Anaheim, CA., http://www.usenix.org. OSF RI papers (Spec ‘93): OSF Mach Final Draft Kernel Interfaces Bibliography Mach 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 177OSF Mach Final Draft Kernel Principles OSF Mach Final Draft Server Library Interfaces OSF Mach Final Draft Server Writer's Guide OSF Mach Kernel Interface Changes OSF RI papers (Spec ‘94): OSF RI 1994 Mach Kernel Interfaces Draft OSF RI 1994 Mach Kernel Interfaces Draft (Part A) OSF RI 1994 Mach Kernel Interfaces Draft (Part B) OSF RI 1994 Mach Kernel Interfaces Draft (Part C) OSF RI papers (miscellaneous): Debugging an object oriented system using the Mach interface Unix File Access and Caching in a Multicomputer Environment Untyped MIG: The Protocol Untyped MIG: What Has Changed and Migration Guide Towards a World-Wide Civilization of Objects A Preemptible Mach Kernel A Trusted, Scalable, Real-Time Operating System Environment Mach Scheduling Framework Networking UNIX Network Programming . Volume 1, Networking APIs: Sockets and XTI . W. Richard Stevens. Prentice Hall, 1998, ISBN 0-13-490012-X. UNIX Network Programming . Volume 2, Interprocess Communications. W. Richard Stevens. Prentice Hall, 1998. ISBN 0-13-081081-9. TCP/IP Illustrated . Volume 1, The Protocols. W. Richard Stevens. Addison-Wesley, 1994. ISBN 0-201-63346-9. Bibliography Networking 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 178TCP/IP Illustrated . Volume 2, The Implementation .W. Richard Stevens.Addison-Wesley, 1995. ISBN0-201-63354-X. TCP/IP Illustrated . Volume 3, TCP for Transactions, HTTP, NNTP, and the UNIX Domain Protocols. W. Richard Stevens. Addison-Wesley, 1996. ISBN 0-201-63495-3. Operating Systems Advanced Computer Architecture: Parallelism, Scalability, Programmability . Kai Hwang. McGraw-Hill, 1993. ISBN 0-07-031622-8. Concurrent Systems: An Integrated Approach to Operating Systems, Database, and Distributed Systems. Jean Bacon. Addison-Wesley, 1993. ISBN 0-201-41677-8. Distributed Operating Systems. Andrew S. Tanenbaum. Prentice Hall, 1995. ISBN 0-13-219908-4. Distributed Operating Systems: The Logical Design . A. Goscinski. Addison-Wesley, 1991. ISBN 0-201-41704-9. Distributed Systems, Concepts, and Designs. G. Coulouris, et al. Addison-Wesley, 1994. ISBN 0-201-62433-8. Operating System Concepts. 4th Ed., Abraham Silberschatz, Peter Galvin. Addison-Wesley, 1994. ISBN 0-201-50480-4. POSIX Information Technology-PortableOperating SystemInterface (POSIX): SystemApplication ProgramInterface (API) (C Language). ANSI/IEEE Std. 1003.1. 1996 Edition. ISO/IEC 9945-1: 1996. IEEE Standards Office. ISBN 1-55937-573-6. Programming with POSIX Threads. David R. Butenhof. Addison Wesley Longman, Inc., 1997. ISBN 0-201-63392-2. Programming Advanced Programming in theUNIX Environment. RichardW. Stevens.Addison-Wesley, 1992. ISBN0-201-56317-7. Debugging with GDB: The GNU Source-Level Debugger Eighth Edition for GDB version 5.0 . Richard Stallman et al. Cygnus Support. http://developer.apple.com/documentation/DeveloperTools/gdb/gdb/gdb_toc.html. Open Source Development with CVS , Karl Franz Fogel. Coriolis Group, 1999. ISBN: 1-57610-490-7. Porting UNIX Software: From Download to Debug . Greg Lehey. O’Reilly, 1995. ISBN 1-56592-126-7. Bibliography Operating Systems 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 179The Standard C Library . P.J. Plauger. Prentice Hall, 1992. ISBN 0-13-131509-9. Websites and Online Resources Apple’s developer website (http://www.apple.com/developer/) is a general repository for developer documentation. Additionally, the following sites provide more domain-specific information. Apple’s Public Source projects and Darwin http://www.opensource.apple.com/ The Berkeley Software Distribution (BSD) http://www.FreeBSD.org http://www.NetBSD.org http://www.OpenBSD.org BSD Networking http://www.kohala.com/start/ Embedded C++ http://www.caravan.net/ec2plus GDB, GNUPro Toolkit 99r1 Documentation http://www.redhat.com/docs/manuals/gnupro/ The Internet Engineering Task Force (IETF) http://www.ietf.org jam http://www.perforce.com/jam/jam.html The PowerPC CPU http://www.freescale.com/webapp/sps/site/homepage.jsp?nodeId=0162468rH3bTdG The Single UNIX Specification Version 2 http://www.opengroup.org/onlinepubs/007908799 Bibliography Websites and Online Resources 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 180The USENIX Association; USENIX Proceedings http://www.usenix.org http://www.usenix.org/publications/library/ Security and Cryptography Applied Cryptography: Protocols, Algorithms, and Source Code in C. Bruce Schneier. John Wiley & Sons, 1994. ISBN 0-471-59756-2. comp.security newsgroup (news:comp.security). comp.security.unix newsgroup (news:comp.security.unix). Computer Security . Dieter Gollmann. John Wiley and Son Ltd, 1999. ISBN 0-471-97844-2. Foundations of Cryptography . Oded Goldreich. Cambridge University Press, 2001. ISBN 0-521-79172-3. Secrets and Lies: Digital Security in a Networked World . Bruce Schneier. John Wiley & Sons, 2000. ISBN 0-471-25311-1. Bibliography Security and Cryptography 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 181This table describes the changes to Kernel Programming Guide . Date Notes 2012-02-16 Updated for OS X v10.7. Added a chapter that discusses the early stages of the boot process. “The Early Boot Process” (page 21) was formerly part of Daemons and Services Programming Guide , and was moved here during a reorganization of that book. 2011-03-08 2006-11-07 Added security information and improved kernel build instructions. 2006-10-03 Made minor corrections. 2006-05-23 Added a note about pmuflags to the debugging section. 2006-04-04 Removed out-of-date information for OS X v10.4. 2006-03-08 Updated some stale content for OS X version 10.4. 2006-01-10 Corrected locking prototypes. Made minor fixesto the file system section. Revised networking, synchronization, and kernel services APIs for OS X v10.4. 2005-11-09 Changed terminology from "fat binary" to "universal binary." Clarified the distinction between memory objects and VM objects. 2005-08-11 2005-07-07 Fixed minor errors in build instructions. 2005-06-04 Updated kernel build instructions for OS X v10.4; other minor fixes. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 182 Document Revision HistoryDate Notes Added information about generating NMI on newer hardware in OS X v10.4 and later; various other minor changes. 2005-04-29 2005-02-03 Made minor corrections. 2004-12-02 Made section number fix to man page reference in Chapter 14. 2004-11-01 Minor wording changes. Added details comparing current_task to mach_task_self. Added information about using AltiVec and floating point in the kernel. 2004-08-01 2003-09-01 Minor corrections to kernel build information Added information relating to Power Macintosh G5 VM issues and debugging. Clarified wait queue documentation (event_t). 2003-08-01 Minor update release. Added index and tweaked wording throughout. Fixed minor errata in debugging chapter. Added a few missing details in the security chapter and cleaned up the equations presented. Corrected a few very minor OS X v10.2-specific details that weren’t caught during the first revision. 2003-02-01 OS X v10.2 update release. Changed information on KEXT management, various small corrections (mainly wording improvements). 2002-08-01 Full web release to coincide with WWDC. Corrected a few minor errata from the previous release. 2002-06-01 2002-01-01 Initial partial web release. Document Revision History 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 183abstraction (v) The process of separating the interface to some functionality from the underlying implementation in such a way that the implementation can be changed without changing the way that piece of code is used. (n) The API (interface) for some piece of functionality that has been separated in this way. address space The virtual addressranges available to a given task (note: the task may be the kernel). In OS X, processes do not share the same address space. The addressspaces of multiple processes can, however, point to the same physical addressranges. This is referred to as shared memory. anonymous memory Virtual memory backed by the default pager to swap files, rather than by a persistent object. Anonymous memory is zero-initialized and exists only for the life of the task. See also default pager; task. API (application programming interface) The interface (calling convention) by which an application program accesses a service. This service may be provided by the operating system, by libraries, or by other parts of the application. Apple Public Source License Apple’s Open Source license, available at http://www.apple.com/publicsource. Darwin is distributed under this license. See also Open Source. AppleTalk A suite of network protocols that is standard on Macintosh computers. ASCII (American Standard Code for Information Interchange) A 7-bit character set (commonly represented using 8 bits) that defines 128 unique character codes. See also Unicode. BSD (Berkeley Software Distribution Formerly known as the Berkeley version of UNIX, BSD is now simply called the BSD operating system. The BSD portion of the OS X kernel is based on FreeBSD, a version of BSD. bundle A directory thatstores executable code and the software resources related to that code. Applications, plug-ins, and frameworks represent types of bundles. Except for frameworks, bundles are presented by the Finder as if they were a single file. Carbon An application environment in OS X that features a set of programming interfaces derived from earlier versions of the Mac OS. The Carbon APIs have been modified to work properly with OS X. Carbon applications can run in OS X, Mac OS 9, and all versions of Mac OS 8 later than Mac OS 8.1 (with appropriate libraries). Classic An application environment in OS X that lets users run non-Carbon legacy Mac OS software. It supports programs built for both Power PC and 68K processor architectures. clock An object used to abstract time in Mach. 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 184 GlossaryCocoa An advanced object-oriented development platform on OS X. Cocoa is a set of frameworks with programming interfaces in both Java and Objective-C. It is based on the integration of OPENSTEP, Apple technologies, and Java. condition variable Essentially a wait queue with additional locking semantics. When a thread sleeps waiting for some event to occur, it releases a related lock so that another thread can cause that event to occur. When the second thread posts the event, the first thread wakes up, and, depending on the condition variable semantics used, either takes the lock immediately or begins waiting for the lock to become available. console (1) A text-based login environment that also displays system log messages, kernel panics, and other information. (2) A special window in OS X that displays messages that would be printed to the text console if the GUI were not in use. This window also displays output written to the standard error and standard output streams by applications launched from the Finder. (3) An application by the same name that displays the console window. control port In Mach, access to the control port allows an object to be manipulated. Also called the privileged port. See also port; name port. cooperative multitasking A multitasking environment in which a running programcan receive processing time only if other programs allow it; each application must give up control of the processor cooperatively in order to allow others to run. Mac OS 9 is a cooperative multitasking environment. See also preemptive multitasking. copy-on-write A delayed copy optimization used in Mach. The object to be copied is marked temporarily read-only. When a thread attempts to write to any page in that object, a trap occurs, and the kernel copies only the page or pages that are actually being modified. See also thread. daemon A long-lived process, usually without a visible user interface, that performs a system-related service. Daemons are usually spawned automatically by the system and may either live forever or be regenerated at intervals. They may also be spawned by other daemons. Darwin The core of OS X, Darwin is an Open Source project that includes the Darwin kernel, the BSD commands and C libraries, and several additional features.The Darwin kernel is synonymous with the OS X kernel. default pager In Mach, one of the built-in pagers. The default pager handles nonpersistent (anonymous)memory. See also anonymousmemory; vnode pager; pager. demand paging An operating-system facility that brings pages of data from disk into physical memory only as they are needed. DLIL (Data Link Interface Layer) The part of the OS X kernel’s networking infrastructure that provides the interface between protocol handling and network device driversin the I/O Kit. A generalization of the BSD “ifnet” architecture. DMA (direct memory access) A means of transferring data between host memory and a peripheral device without requiring the host processor to move the data itself. This reduces processor overhead for I/O operations and may reduce contention on the processor bus. Glossary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 185driver Software that deals with getting data to and from a device, as well as control of that device. In the I/O Kit, an object that manages a piece of hardware (a device), implementing the appropriate I/O Kit abstractions for that device. See also object. DVD (Digital Versatile Disc) Originally, Digital Video Disc. An opticalstorage medium that provides greater capacity and bandwidth than CD-ROM; DVDs are frequently used for multimedia as well as data storage. dyld (dynamic link editor) A utility that allows programs to dynamically load (and link to) needed functions. EMMI (External Memory Management Interface) Mach’sinterface to memory objectsthat allowstheir contents to be contributed by user-mode tasks. See also external pager. Ethernet A family of high-speed local area network technologies in common use. Some common variants include 802.3 and 802.11 (Airport). exception An interruption to the normal flow of program control, caused by the program itself or by executing an illegal instruction. exception port A Mach port on which a task or thread receives messages when exceptions occur. external pager A module that manages the relationship between virtual memory and a backing store. External pagers are clients of Mach’s EMMI. The pager API is currently not exported to userspace. The built-in pagersin OS X are the default pager, the device pager, and the vnode pager. See also EMMI (External Memory Management Interface). family In the I/O Kit, a family defines a collection of software abstractions that are common to all devices of a particular category (for example, PCI, storage, USB). Families provide functionality and services to drivers. See also driver. FAT (file allocation table) A data structure used in the MS-DOS file system. Also synonymous with the file system that uses it. The FAT file system is also used as part of Microsoft Windows and has been adopted for use inside devices such as digital cameras. fat files See universal binaries. FIFO (first-in first-out) A data processing scheme in which data is read in the order in which it was written, processes are run in the order in which they were scheduled, and so forth. file descriptor A per-process unique, nonnegative integer used to identify an open file (or socket). firewall Software (or a computer running such software) that prevents unauthorized access to a network by users outside of the network. fixed-priority policy In Mach, a scheduling policy in which threads execute for a certain quantum of time, and then are put at the end of the queue of threads of equal priority. fork (1) A stream of data that can be opened and accessed individually under a common filename. The Macintosh Standard and Extended file systems store a separate “data” fork and a “resource” fork as part of every file; data in each fork can be accessed and manipulated independently of the other. (2) In BSD, fork is a system call that creates a new process. framework A bundle containing a dynamic shared library and associated resources, including image files, header files, and documentation. Frameworks Glossary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 186are often used to provide an abstraction for manipulating device driver families from applications. FreeBSD A variant of the BSD operating system. See http://www.freebsd.org for details. gdb (GNU debugger) gdb is a powerful, source-level debugger with a command-line interface. gdb is a popular Open Source debugger and is included with the OS X developer tools. HFS (hierarchical file system) The Mac OS Standard file system format, used to represent a collection of files as a hierarchy of directories (folders), each of which may contain either files or foldersthemselves. HFS+ The Mac OS Extended file system format. This file system format was introduced as part of Mac OS 8.1, adding support for filenames longer than 31 characters, Unicode representation of file and directory names, and efficient operation on larger disks. host (1) The computer that is running (is host to) a particular program or service. The term is usually used to refer to a computer on a network. (2) In debugging, the computer that is running the debugger itself. In this context, the target is the machine running the application, kernel, or driver being debugged. host processor The microprocessor on which an application program resides. When an application is running, the host processor may call other, peripheral microprocessors, such as a digital signal processor, to perform specialized operations. IDE (integrated development environment) An application or set of tools that allows a programmer to write, compile, edit, and in some cases test and debug within an integrated, interactive environment. inheritance attribute In Mach, a value indicating the degree to which a parent process and its child process share pages in the parent process’s address space. A memory page can be inherited as copy-on-write, shared, or not at all. in-line data Data that’s included directly in a Mach message, rather than referred to by a pointer. See also out-of-line data. info plist See information property list. information property list A special form of property list with predefined keysforspecifying basic bundle attributes and information of interest, such as supported document types and offered services. See also bundle; property list. interrupt service thread A thread running in kernel space for handling I/O that is triggered by an interrupt, but does not run in an interrupt context. Also called an I/O service thread. I/O (input/output) The exchange of data between two parts of a computer system, usually between system memory and a peripheral device. I/O Kit Apple’s object-oriented I/O development model. The I/O Kit provides a framework for simplified driver development, supporting many families of devices. See also family. I/O service thread See interrupt service thread. IPC (interprocess communication) The transfer of information between processes or between the kernel and a process. IPL (interrupt priority level) A means of basic synchronization on uniprocessor systems in traditional BSD systems, set using the spl macro. Interrupts with lower priority than the current IPL will not be acted upon until the IPL is lowered. In many parts of the kernel, changing the IPL in OS X Glossary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 187is not useful as a means ofsynchronization. New use of spl macros is discouraged. See also spl (set priority level). KDP The kernelshim used for communication with a remote debugger (gdb). Kerberos An authentication system based on symmetric key cryptography. Used in MIT Project Athena and adopted by the Open Software Foundation (OSF). kernel The complete OS X core operating-system environment that includes Mach, BSD, the I/O Kit, file systems, and networking components. kernel crash An unrecoverable system failure in the kernel caused by an illegal instruction, memory access exception, or other failure rather than explicitly triggered as in a panic. See also panic. kernel extension See KEXT (kernel extension). kernel mode See supervisor mode. kernel panic See panic. kernel port A Mach port whose receive right is held by the kernel. See also task port; thread port. KEXT (kernel extension) A bundle that extendsthe functionality of the kernel. The I/O Kit, File system, and Networking components are designed to allow and expect the creation and use of KEXTs. KEXT binary A file (or files) in Mach-O format, containing the actual binary code of a KEXT. A KEXT binary is the minimum unit of code that can be loaded into the kernel. Also called a kernel module or KMOD. See also KEXT (kernel extension); Mach-O. key signing In public key cryptography, to (electronically)state your trust that a public key really belongs to the person who claims to own it, and potentially that the person who claims to own it really is who he or she claims to be. KMOD (kernel module) See KEXT binary. lock A basic means of synchronizing multiple threads. Generally only one thread can “hold” a lock at any given time. While a thread is holding the lock, any other thread that tries to take it will wait, either by blocking or by spinning, depending on the nature of the lock. Some lock variants such as read-write locks allow multiple threads to hold a single lock under certain conditions. Mach The lowest level of the OS X kernel. Mach provides such basic services and abstractions as threads, tasks, ports, IPC, scheduling, physical and virtual address space management, VM, and timers. Mach-O Mach object file format. The preferred object file format for OS X. Mach server A task that providesservicesto clients, using a MIG-generated RPC interface. See also MIG (Mach interface generator). main thread By default, a process has one thread, the main thread. If a process has multiple threads, the main thread is the first thread in the process. A user process can use the POSIX thread API to create other user threads. makefile A makefile detailsthe files, dependencies, and rules by which an executable application is built. memory-mapped files A facility that maps virtual memory onto a physical file. Thereafter, any access to that part of virtual memory causes the corresponding page of the physical file to be accessed. The contents of the file can be changed by changing the contents in memory. Glossary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 188memory object An object managed by a pager that represents the memory, file, or other storage that backs a VM object. See also pager. memory protection A system of memory management in which programs are prevented from being able to modify or corrupt the memory partition of another program, usually through the use of separate address spaces. message A unit of data sent by one task or thread that is guaranteed to be delivered atomically to another task or thread. In Mach, a message consists of a header and a variable-length body. Some system services are invoked by passing a message from a thread to the Mach port representing the task that provides the desired service. microkernel A kernel implementing a minimal set of abstractions. Typically, higher-level OS services such as file systems and device drivers are implemented in layers above a microkernel, possibly in trusted user-mode servers. OS X is a hybrid between microkernel and monolithic kernel architectures. See also monolithic kernel. MIG (Mach interface generator) (1) A family of software that generates and supports the use of a procedure call interface to Mach’s system of interprocess communication. (2) The interface description language supported by MIG. monolithic kernel A kernel architecture in which all pieces of the kernel are closely intertwined. A monolithic kernel providessubstantial performance improvements. It is difficult to evolve the individual components independently, however. The OS X kernel is a hybrid of the monolithic and microkernel models. See also microkernel. multicast A process in which a single packet can be addressed to multiple recipients. Multicast is used, for example, in streaming video, in which many megabytes of data are sent over the network. multihoming The ability to have multiple network addresses in one computer, usually on different networks. For example, multihoming might be used to create a system in which one address is used to talk to hosts outside a firewall and the other to talk to hosts inside; the computer provides facilities for passing information between the two. multitasking The concurrent execution of multiple programs. OS X uses preemptive multitasking. Mac OS 9 uses cooperative multitasking. mutex See mutex lock (mutual exclusion lock). mutex lock (mutual exclusion lock) A type of lock characterized by putting waiting threads to sleep until the lock is available. named (memory) entry A handle (a port) to a mappable object backed by a memory manager. The object can be a region or a memory object. name port In Mach, accessto the name port allows non-privileged operations against an object (for example, obtaining information about the object). In effect, it provides a name for the object without providing any significant access to the object. See also port; control port. named region In Mach, a form of named memory entry that provides a form of memory sharing. namespace An agreed-upon context in which names (identifiers) can be defined. Within a given namespace, all names must be unique. Glossary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 189NAT (network address translation) A scheme that transforms network packets at a gateway so network addresses that are valid on one side of the gateway are translated into addresses that are valid on the other side. network A group of hosts that can communicate with each other. NFS (network file system) A commonly used file server protocol often found in UNIX and UNIX-based environments. NKE (network kernel extension) A type of KEXT that provides a way to extend and modify the networking infrastructure of OS X dynamically without recompiling or relinking the kernel. NMI (nonmaskable interrupt) An interrupt produced by a particular keyboard sequence or button that cannot be blocked in software. It can be used to interrupt a hung system, for example to drop into a debugger. nonsimple message In Mach, a message that contains either a reference to a port or a pointer to data. See also simple message. notify port A special Mach port that is part of a task. A task’s notify port receives messages from the kernel advising the task of changes in port access rights and of the status of messages it has sent. nub An I/O Kit object that represents a point of connection for a device or logical service. Each nub provides accessto the device orservice it represents, and provides such services as matching, arbitration, and power management. It is most common that a driver publishes one nub for each individual device or service it controls; it is possible for a driver that vends only a single device orservice to act asits own nub. NVRAM (nonvolatile RAM) RAM storage that retains its state even when the power is off. See also RAM (random-access memory). object (1) A collection of data. (2) In Mach, a collection of data, with permissions and ownership. (3) In object-oriented programming, an instance of a class. OHCI (Open Host Controller Interface) The register-level standards that are used by most USB and Firewire controller chips. Open Source Software that includesfreely available access to source code, redistribution, modification, and derived works. The full definition is available at http://www.opensource.org. Open Transport A communications architecture for implementing network protocols and other communication features on computers running classic Mac OS. Open Transport provides a set of programming interfacesthatsupports, among other things, both the AppleTalk and TCP/IP protocols. out-of-line data Data that’s passed by reference in a Mach message, rather than being included in the message. See also in-line data. packet An individual piece of information sent on a network. page (n) (1) The largest block of virtual address space for which the underlying physical address space is guaranteed contiguous—in other words, the unit of mapping between virtual and physical addresses. (2) logical page size: The minimum unit of information that an anonymous pager transfers between system memory and the backing store. (3) physical page size: The unit of information treated as a unit by a hardware MMU. The logical page size must be at least as large as the physical page size Glossary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 190for hardware-based memory protection to be possible. (v) To move data between memory and a backing store. pager A module responsible for providing the data for the pages of a memory object. See also default pager; vnode pager. panic An unrecoverable system failure explicitly triggered by the kernel with a call to panic. See also kernel crash. PEF (Preferred Executable Format) The format of executable files used for applications and shared libraries in Mac OS 9; supported in OS X. The preferred format for OS X is Mach-O. physical address An address to which a hardware device,such as a memory chip, can directly respond. Programs, including the Mach kernel, use virtual addresses that are translated to physical addresses by mapping hardware controlled by the Mach kernel. pmap Part of Mach VM that provides an abstract way to set and fetch virtual to physical mappings from hardware. The pmap system is the machine-dependent layer of the VM system. port In Mach, a secure unidirectional channel for communication between tasks running on a single system. In IP transport protocols, an integer identifier used to select a receiving service for an incoming packet, or to specify the sender of an outgoing packet. port name In Mach, an integer index into a port namespace; a port right is specified with respect to its port name. See also port rights. portrights In Mach, the ability to send to or receive from a Mach port. Also known as port access rights. port set In Mach, a set of zero or more Mach ports. A thread can receive messages sent to any of the ports contained in a port set by specifying the port set as a parameter to msg_receive(). POSIX (Portable Operating System Interface) A standard that defines a set of operating-system services. It is supported by ISO/IEC, IEEE, and The Open Group. preemption The act of interrupting a currently running program in order to give time to another task. preemptive multitasking A type of multitasking in which the operating system can interrupt a currently running task in order to run another task, as needed. See also cooperative multitasking. priority In scheduling, a number that indicates how likely a thread is to run. The higher the thread’s priority, the more likely the thread isto run. See also scheduling policy. process A BSD abstraction for a running program. A process’s resources include an address space, threads, and file descriptors. In OS X, a process is based on one Mach task and one or more Mach threads. process identifier (PID), A number that uniquely identifies a process. Also called a process ID. programmed I/O I/O in which the CPU accomplishes data transfer with explicit load and store instructions to device registers, rather than DMA, and without the use of interrupts. This data transfer is often done in a byte-by-byte, or word-by-word fashion. Also known as direct or polled I/O. See also DMA (direct memory access). property list A textual way to represent data. Elements of the property list represent data of certain types,such as arrays, dictionaries, and strings. Glossary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 191System routines allow programs to read property lists into memory and convert the textual data representation into “real” data. See also information property list. protected memory See memory protection. protocol handler A network module that extracts data from input packets (giving the data to interested programs) and inserts data into output packets(giving the output packet to the appropriate network device driver). pthreads The POSIX threads implementation. See also POSIX (Portable Operating System Interface); thread. quantum The fixed amount of time a thread or process can run before being preempted. RAM (random-access memory) Memory that a microprocessor can either read from or write to. real-time performance Performance characterized by guaranteed worst-case response times. Real-time support is important for applications such as multimedia. receive rights In Mach, the ability to receive messages on a Mach port. Only one task at a time can have receive rights for any one port. See also send rights. remote procedure call See RPC (remote procedure call). reply port A Mach port associated with a thread that is used in remote procedure calls. ROM (read-only memory) Memory that cannot be written to. root (1) An administrative account with special privileges. For example, only the root account can load kernel extensions.(2) In graph theory, the base of a tree. (3) root directory: The base of a file system tree. (4) root file system: The primary file system off which a computer boots, so named because it includes the root node of the file system tree. routine In Mach, a remote procedure call that returns a value. This can be used for synchronous or asynchronous operations. See also simpleroutine. RPC (remote procedure call) An interface to IPC that appears (to the caller) as an ordinary function call. In Mach, RPCs are implemented using MIG-generated interface libraries and Mach messages. scheduling The determination of when each process or task runs, including assignment of start times. scheduling policy In Mach, how the thread’s priority isset and under what circumstancesthe thread runs. See also priority. SCSI (Small Computer Systems Interface) A standard communications protocol used for connecting devicessuch as disk drivesto computers. Also, a family of physical bus designs and connectors commonly used to carry SCSI communication. semaphore Similar to a lock, except that a finite number of threads can be holding a semaphore at the same time. See also lock. send rights In Mach, the ability to send messages to a Mach port. Many tasks can have send rights for the same port. See also receive rights. session key In cryptography, a temporary key that is only used for one message, one connection session, orsimilar. Session keys are generally treated asshared secrets, and are frequently exchanged over a channel encrypted using public key cryptography. Glossary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 192shadow object In Mach VM, a memory object that holds modified pages that originally belonged to another memory object. Thisis used when an object that was duplicated in a copy-on-write fashion is modified. If a page is not found in this shadow object, the original object is referenced. simple message In Mach, a message that contains neither references to ports nor pointers to data. See also nonsimple message. simpleroutine In Mach, a remote procedure call that does not return a value, and has no out or inout parameters. This can be used for asynchronous operations. See also routine. SMP (symmetric multiprocessing) A system architecture in which two or more processors are managed by one kernel, share the same memory, have equal access to I/O devices, and in which any task, including kernel tasks, can run on any processor. spinlock Any of a family of lock types characterized by continuously polling to see if a lock is available, rather than putting the waiting thread to sleep. spin/sleep lock Any of a family of lock types characterized by some combination of the behaviors of spinlocks and mutex (sleep) locks. spl (set priority level) A macro thatsetsthe current IPL. Interrupts with lower priority than the current IPL will not be acted upon until the IPL is lowered. The spl macros have no effect in many parts of OS X, so their use is discouraged as a means of synchronization in new programming except when modifying code that already uses spl macros. See also IPL (interrupt priority level). socket (1) In a user process, a file descriptor that has been allocated using socket(2). (2) In the kernel, the data structure allocated when the kernel’s implementation of the socket(2) call is made. (3) In AppleTalk protocols, a socket serves the same purpose as a port in IP transport protocols. submap A collection of mappingsin the VM system that is shared among multiple Mach tasks. supervisor mode Also known as kernel mode, the processor mode in which certain privileged instructions can be executed, including those related to page table management, cache management, clock setting, and so on. symmetric multiprocessing See SMP (symmetric multiprocessing). task A Mach abstraction, consisting of a virtual address space and a port namespace. A task itself performs no computation; rather, it isthe framework in which threads run. See also thread. task port A kernel port that represents a task and is used to manipulate that task. See also kernel port; thread port. TCP/IP (Transmission Control Protocol/Internet Protocol) An industry standard protocol used to deliver messages between computers over the network. TCP/IP is the primary networking protocol used in OS X. thread The unit of program execution. A thread consists of a program counter, a set of registers, and a stack pointer. See also task. thread port A kernel port that represents a thread and is used to manipulate that thread. See also kernel port; task port. thread-safe code Code that can be executed safely by multiple threads simultaneously. Glossary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 193time-sharing policy In Mach, a scheduling policy in which a thread’s priority is raised and lowered to balance its resource consumption against other timesharing threads. UDF (Universal Disk Format) The file system format used in DVD disks. UFS (UNIX file system) An industry standard file system format used in UNIX and similar operating systems such as BSD. UFS in OS X is a derivative of 4.4BSD UFS. Unicode A 16-bit character set that defines unique character codes for characters in a wide range of languages. Unlike ASCII, which defines 128 distinct characters typically represented in 8 bits, there are as many as 65,536 distinct Unicode characters that represent the unique characters used in most foreign languages. universal binaries Executable files containing object code for more than one machine architecture. UPL (universal page list) A data structure used when communicating with the virtual memory system. UPLs can be used to change the behavior of pages with respect to caching, permissions, mapping, and so on. USB (Universal Serial Bus) A multiplatform bus standard that can support up to 127 peripheral devices, including printers, digital cameras, keyboards and mice, and storage devices. UTF-8 (Unicode Transformation Format 8) A format used to represent a sequence of 16-bit Unicode characters with an equivalent sequence of 8-bit characters, none of which are zero. This sequence of characters can be represented using an ordinary C language string. VFS (virtual file system) A set of standard internal file-system interfaces and utilities that facilitate support for additional file systems. VFS provides an infrastructure for file systems built into the kernel. virtual address An address as viewed from the perspective of an application. Each task has its own range of virtual addresses, beginning at address zero. The Mach VM system makes the CPU hardware map these addresses onto physical memory. See also physical address. virtual memory A system in which addresses as seen by software are not the same as addressesseen by the hardware. This provides support for memory protection, reduces the need for code relocatability, and allows the operating system to provide the illusion to each application that it has resources much larger than those that could actually be backed by RAM. VM See virtual memory. vnode An in-memory data structure containing information about a file. vnode pager In Mach, one of the built-in pagers. The vnode pager maps files into memory objects. See also default pager; pager. work loop The main loop of an application or KEXT that waits repeatedly for incoming events and dispatches them. XML (Extensible Markup Language) A dialect of SGML (Standard Generalized Markup Language), XML provides a metalanguage containing rules for constructing specialized markup languages. XML users can create their own tags, making XML very flexible. Glossary 2012-02-16 | © 2002, 2012 Apple Inc. All Rights Reserved. 194Apple Inc. © 2002, 2012 Apple Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrievalsystem, or transmitted, in any form or by any means, mechanical, electronic, photocopying, recording, or otherwise, without prior written permission of Apple Inc., with the following exceptions: Any person is hereby authorized to store documentation on a single computer for personal use only and to print copies of documentation for personal use provided that the documentation contains Apple’s copyright notice. No licenses, express or implied, are granted with respect to any of the technology described in this document. Apple retains all intellectual property rights associated with the technology described in this document. This document is intended to assist application developers to develop applications only for Apple-labeled computers. Apple Inc. 1 Infinite Loop Cupertino, CA 95014 408-996-1010 Apple, the Apple logo, AppleTalk, Carbon, Cocoa, Finder, FireWire, Keychain, Logic, Mac, Mac OS, Macintosh, Objective-C, OS X, Pages, Panther, Power Mac,Quartz,QuickTime, Spaces, and Xcode are trademarks of Apple Inc., registered in the U.S. and other countries. .Mac is a service mark of Apple Inc., registered in the U.S. and other countries. NeXT and OPENSTEP are trademarks of NeXT Software, Inc., registered in the U.S. and other countries. DEC is a trademark of Digital Equipment Corporation. Intel and Intel Core are registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Java is a registered trademark of Oracle and/or its affiliates. OpenGL is a registered trademark of Silicon Graphics, Inc. PowerPC and the PowerPC logo are trademarks of International Business Machines Corporation, used under license therefrom. SPEC is a registered trademark of the Standard Performance Evaluation Corporation (SPEC). UNIX is a registered trademark of The Open Group. Even though Apple has reviewed this document, APPLE MAKES NO WARRANTY OR REPRESENTATION, EITHER EXPRESS OR IMPLIED, WITH RESPECT TO THIS DOCUMENT, ITS QUALITY, ACCURACY, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.ASARESULT, THISDOCUMENT IS PROVIDED “AS IS,” AND YOU, THE READER, ARE ASSUMING THE ENTIRE RISK AS TO ITS QUALITY AND ACCURACY. IN NO EVENT WILL APPLE BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL,OR CONSEQUENTIAL DAMAGES RESULTING FROM ANY DEFECT OR INACCURACY IN THIS DOCUMENT, even if advised of the possibility of such damages. THE WARRANTY AND REMEDIES SET FORTH ABOVE ARE EXCLUSIVE AND IN LIEU OF ALL OTHERS, ORAL OR WRITTEN, EXPRESS OR IMPLIED. No Apple dealer, agent, or employee is authorized to make any modification, extension, or addition to this warranty. Some states do not allow the exclusion or limitation of implied warranties or liability for incidental or consequential damages, so the above limitation or exclusion may not apply to you. This warranty gives you specific legal rights, and you may also have other rights which vary from state to state. RED Workflows with Final Cut Pro X White Paper June 2012White Paper 2 RED Workflows with Final Cut Pro X With the continuing popularity of the RED® family of cameras (www.red.com), Final Cut Pro X editors have been looking for proven workflows with REDCODE® RAW files. This white paper outlines how professional production companies are achieving excellent results when recording with RED cameras, editing in Final Cut Pro X, and finishing in applications such as DaVinci Resolve. This document outlines a complete RED-based post-production workflow, following the steps below: 1. Transcode REDCODE RAW files to Apple ProRes using REDCINE-X® PRO. 2. Batch sync audio and video files. 3. Import synced files into Final Cut Pro X. During import, Final Cut Pro X can automatically create lightweight Apple ProRes 422 (Proxy) files for editing. Or, if you have a lot of footage and multiple editors, you can use Compressor to create the Apple ProRes 422 (Proxy) files. 4. Edit and lock picture with Final Cut Pro X. 5. Export an XML file of the project from Final Cut Pro X. 6. Color grade the project in DaVinci Resolve using either high-quality Apple ProRes or R3D RAW files. You can relink the project XML file to the original R3D files in either REDCINE-X PRO or DaVinci Resolve. 7. Export an XML file from DaVinci Resolve and import it back into Final Cut Pro X. 8. Export a final master from Final Cut Pro X. This method combines the best of both worlds—the speed of editing with Apple ProRes on a wide variety of notebook and desktop systems, and the color grading advantages of RAW when finishing. You can further simplify this workflow by transcoding to high-quality Apple ProRes files and using those throughout color grading and delivery. The sections below include additional detail about each stage of the workflow. Transcode REDCODE RAW Files to Apple ProRes RED cameras record a RAW file (R3D) that must be “debayered” and decompressed to convert the original sensor data to viewable pixels, so that the file can play back in video editing software. Apple ProRes is an excellent choice for this conversion, because it’s a codec optimized for both quality and editing speed. Apple ProRes is a full frame size, intra-frame codec designed to efficiently use multiple processors for playback and rendering. The free RED application REDCINE-X PRO supports Apple ProRes encoding, which can be accelerated using the RED ROCKET® card. REDCINE-X PRO also allows you to apply a “one-light” color correction during the transcoding process, giving your footage a more finished look for editing and review.White Paper 3 RED Workflows with Final Cut Pro X In the Field As workflows expand to include field review and even rough cut editing, digital imaging technicians (DITs) are actively transcoding R3D footage to Apple ProRes using high-end portable systems. A typical field-based workflow includes these steps: • Copy the footage from the camera’s recording medium, like the REDMAG™ removable solid-state drive (SSD). • Create backups so that camera originals can be stored in two different places for data protection. • Create Apple ProRes dailies by transcoding the R3D files to Apple ProRes files for editing and H.264 files for uploading to a secure website for client review. Alternatively, after making backup copies, you can deliver the R3D files to the post facility for transcoding to Apple ProRes dailies. Choose Transcoding Settings When you transcode your files to Apple ProRes, choose the level of quality that’s appropriate to your specific production. Workflow Apple ProRes codec Disk space is a consideration, or you’re editing a large multicam project. Apple ProRes 422 (Proxy) or Apple ProRes 422 (LT) You’re delivering Apple ProRes files as a final master for the web or TV. Apple ProRes 422 or Apple ProRes 422 (HQ) You’re delivering for theater projection or effects compositing. Apple ProRes 4444 Although you can transcode to the final delivery quality and then work with that throughout post-production, it’s more efficient to work with smaller frame sizes and higher image compression during the craft edit. So even though you may have shot at 4K or 5K resolution in the field, you can transcode to a smaller frame size to save time and disk space. For example, you can set the resolution to 1920x1080 or 1280x720, and you can set the debayer quality to 1/4. If you’re generating Apple ProRes files for use as proxy media, you can also choose to superimpose, or “burn in,” the source timecode and filename over the image. This makes it easy to go back to the original R3D files at any point during post-production for a quick visual double-check that the files are correct. For more details, see the REDCINE-X PRO manual at https://www.red.com/downloads. Note: You can speed up transcoding by using a RED ROCKET card, which offloads the processor-intensive debayer and decompression tasks from software to custom hardware. RED ROCKET can be a valuable tool when transcoding a large number of shots.White Paper 4 RED Workflows with Final Cut Pro X Apply One-Light Color Correction When recording with RED cameras, it’s common to shoot a scene “flat” to avoid clipping highlights and shadows and provide more flexibility when manipulating images in post-production. This recording setup can give the footage a washed-out appearance. Many editors and clients prefer working with more visually appealing images that include higher contrast and color saturation. To accommodate this workflow, the free REDCINE-X PRO application allows you to add a one-light color correction as part of the transcoding process. You can choose from several presets to create more common looks, or you can create your own look. Be sure to keep the original names of the R3D files when you generate new Apple ProRes media so that you can easily relink to them later. After you apply the one-light color correction during transcoding, it stays with the image until you go back to the original R3D file and create a new Apple ProRes version. Batch Sync Audio and Video Files After all your media has been transcoded, you can choose to sync second-source audio to the video files. You can sync the files directly in Final Cut Pro X using the built-in synchronization feature that analyzes waveforms to match the scratch audio in your video files to the high-quality audio from your field recorder. You can also use a third-party application like Intelligent Assistance’s Sync-N-Link X (www.intelligentassistance.com), RED’s REDCINE-X PRO (www.red.com/learn), or Singular Software’s PluralEyes (www.singularsoftware.com). Simply select all the audio and Apple ProRes video files and batch sync. Then export the XML to Final Cut Pro X, and all of the synced material is imported into an event, ready for editing. Import Files into Final Cut Pro X After creating Apple ProRes files with REDCINE-X PRO, you can import these files directly into Final Cut Pro X. Even if you transcode R3D files to a high-quality Apple ProRes codec, such as Apple ProRes 4444, you may still choose to use lightweight Apple ProRes 422 (Proxy) files for editing. Final Cut Pro X allows you to generate Apple ProRes proxy files in the background and seamlessly switch to these files for editing, providing great flexibility when editing on a notebook, for example. To create proxy files while importing media 1. In Final Cut Pro, choose File > Import > Files. 2. Select a file or folder, or Command-click to select multiple files to import. 3. Do one of the following: • To add the imported files to an existing event: Select “Add to existing Event,” and choose the event from the pop-up menu. • To create a new event: Select “Create new Event,” type a name in the text field, and choose the disk where you want to store the event from the “Save to” pop-up menu. 4. To have Final Cut Pro copy your media files and add them to the Final Cut Pro Events folder you specified, select the “Copy files to Final Cut Events folder” checkbox. If you’re working with a SAN and want to keep the files in a central location and have multiple users link to them, leave this option unselected. For more information, see Final Cut Pro X: Xsan Best Practices.White Paper 5 RED Workflows with Final Cut Pro X 5. Select the “Create proxy media” checkbox. When this option is selected, Final Cut Pro creates Apple ProRes 422 (Proxy) files in the background after the media files are imported. You can begin to edit your project and, when the proxy files are created, you can open Playback preferences and switch to the proxy files with a single click. 6. Click Import. Final Cut Pro imports your media in the background, and then creates proxy files in the background. You can view the progress of the background tasks in the Background Tasks window. You can now begin editing, even if importing and transcoding are not yet complete. To switch to the Apple ProRes proxy files, select “Use proxy media” in Final Cut Pro Playback preferences. It’s just as easy to switch back to the original media when the creative editing is finished and you want to work on color or effects at the highest quality. When you change these settings, all media in events and projects is affected. Edit in Final Cut Pro X and Export XML After all your media has been imported into Final Cut Pro X, you can edit just as you would any other project. The application was designed for modern, file-based workflows, making it easy to browse, organize, and edit large amounts of material. Use skimming to quickly view your footage. Mark range-based keywords and favorites, and save custom searches as Smart Collections. Quickly and easily arrange clips in the Timeline and add titles and effects, which render in the background as you work. When you’re finished editing, you can send your project to a third-party finishing system such as DaVinci Resolve. Just select the project in the Project Library, choose File > Export XML, and select a location to save the XML file. Color Grade in DaVinci Resolve and Export XML Choose Apple ProRes or RAW for Grading Before importing the Final Cut Pro X XML file into DaVinci Resolve, you should choose between a few different color grading workflows. If you edited with Apple ProRes 422 (HQ) or Apple ProRes 4444 in Final Cut Pro X, you may want to grade these same files in DaVinci Resolve. Alternatively, you can relink the project to the original R3D files in either DaVinci Resolve or REDCINE-X PRO. These RAW files offer a wide range of values to use when grading, which can help improve the look of images that were shot without extensive lighting control or that need a unique style. You can get more image detail out of the highlights and shadows, which is why so many colorists choose to use the RAW files in the color grading stage. White Paper 6 RED Workflows with Final Cut Pro X To relink to the original R3D media in DaVinci Resolve 1. In DaVinci Resolve Preferences, add the location of the R3D files to the Media Storage Volumes list. 2. Save the preferences and reopen DaVinci Resolve. 3. On the Browse page, import the R3D files to the Media Pool. 4. On the Conform page, in the Timeline Management section, click the Load button. 5. Select the XML file that you exported from Final Cut Pro X. 6. In the Load XML window, deselect “Automatically import source clips into media pool.” 7. Choose any other options that are applicable to your project, and click Ok. The XML file is imported and relinked to the corresponding media in the Media Pool using reel name and timecode. A new session appears in the Timeline Management list, and the edit appears in the Timeline. Note: Alternatively, if you’re working with large amounts of media and DaVinci Resolve 8.1 or later, you can have DaVinci Resolve relink to the R3D files automatically when you import the XML file. Just be sure to select the following checkboxes: • Automatically import source clips into media pool • Ignore file extensions when matching Render New Media After color grading the final project in DaVinci Resolve, you can choose the render format based on your final delivery. For example, you may choose to render Apple ProRes 4444 for theater projection, or Apple ProRes 422 if you’re delivering a master for the web or TV. You may want to set a handle length for the rendered media (at least one second), so that you can make additional changes such as adding a longer dissolve or extending an edit. For more details, see the DaVinci Resolve manual at http://www.blackmagic-design.com/support. Export XML from DaVinci Resolve and Import into Final Cut Pro X After you render the media in DaVinci Resolve, you can transfer the project back to Final Cut Pro X by exporting an XML file. To export an XML file from DaVinci Resolve 1. Open the Conform page and, in the Timeline Management list, select the session you want to export an XML file from. 2. Click the Export button at the bottom of the Timeline Management list. 3. In the Export XML dialog, choose FCP X XML 1.1 Files from the Format pop-up menu, type a name and choose a location for the exported XML file, and click Save. An XML version of that session is saved, complete with internal references to the graded media you rendered, and ready for importing into Final Cut Pro X. Import the XML file back into Final Cut Pro X using the Import XML command in the File menu. Make sure that you’re linking to the high-quality media by selecting “Use original or optimized media” in the Playback pane of the Final Cut Pro Preferences window. Now you can add finished audio, adjust titles, insert graphics, and continue to make editorial changes. Because you’ve imported the individual media files and the XML metadata instead of a single QuickTime movie, you can make changes right up to the last minute before delivery. For information about Final Cut Pro X, see Final Cut Pro X Help.White Paper 7 RED Workflows with Final Cut Pro X Copyright © 2012 Apple Inc. All rights reserved. Apple, the Apple logo, Final Cut, Final Cut Pro, QuickTime, and Xsan are trademarks of Apple Inc., registered in the U.S. and other countries. R3D, RED, REDCINE, REDCINE-X, REDCODE, REDMAG, and RED ROCKET are trademarks or registered trademarks of Red.com, Inc. in the United States and other countries. The YouTube logo is a trademark of Google Inc. Other product and company names mentioned herein may be trademarks of their respective companies. Mention of third-party products is for informational purposes only and constitutes neither an endorsement nor a recommendation. Apple assumes no responsibility with regard to the performance or use of these products. Product specifications are subject to change without notice. 019-2378 June 2012 Export a Master from Final Cut Pro X The final step in the workflow is to export a finished master from Final Cut Pro X. To export your project as a master file 1. To make sure the project’s render format is set to the quality level you want for the final master, select the project in the Project Library, click the Inspector button in the toolbar, and click the Project Properties button . The Render Format pop-up menu shows the current render codec. 2. Select the project in the Project Library and choose Share > Export Media (or press Command-E). 3. Choose an option from the Export pop-up menu. The default setting, Video and Audio, creates a movie file containing both video and audio. For information about the other options, see Final Cut Pro X Help. To export a file that matches the project’s properties, choose Current Settings from the “Video codec” pop-up menu. When you export using Current Settings, the final master is exported at the quality of the render settings, and the export is as fast as a file copy with no further compression added. 4. To see details about the file that will be output, click Summary. 5. Click Next, type a name and choose a location for the exported file, and click Save. If you’re exporting for review on the web, you can export an H.264 version directly to a private account on YouTube or Vimeo. You can also burn the project to a DVD, or to a Blu-ray disc if you have a third-party Blu-ray burner. If you have Compressor installed, you can choose Share > Send to Compressor to transfer your project to that application for total control over your final export settings. Compressor also allows you to set up render clusters that use the processors of multiple computers on a network. Create a Compressor droplet for drag-and-drop simplicity, or create custom export settings to match unique delivery requirements. If you need to output to tape, all three major video I/O device manufacturers offer free software to support tape delivery. The applications are AJA‘s VTR Xchange, Blackmagic Design’s Media Express, and Matrox’s Vetura. Download the application that works with your video I/O device and use the QuickTime export from Final Cut Pro X to lay back to tape. Conclusion Using Apple ProRes for editing and R3D RAW files for color grading enables a highly flexible workflow optimized for speed, quality, and creative control. This process also takes advantage of the metadata and XML capabilities of Final Cut Pro X, which have been designed for the future of file-based production. By using this document as a template for working with RED and Final Cut Pro X, editors and post-production facilities can further customize the process to suit their unique needs. Transitioning to ARC Release NotesContents Transitioning to ARC Release Notes 3 Summary 3 ARC Overview 4 ARC Enforces New Rules 5 ARC Introduces New Lifetime Qualifiers 7 ARC Uses a New Statement to Manage Autorelease Pools 11 Patterns for Managing Outlets Become Consistent Across Platforms 11 Stack Variables Are Initialized with nil 12 Use Compiler Flags to Enable and Disable ARC 12 Managing Toll-Free Bridging 12 The Compiler Handles CF Objects Returned From Cocoa Methods 13 Cast Function Parameters Using Ownership Keywords 14 Common Issues While Converting a Project 15 Frequently Asked Questions 19 Document Revision History 23 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 2Automatic Reference Counting (ARC) is a compiler feature that provides automatic memory management of Objective-C objects. Rather than having to think about about retain and release operations, ARC allows you to concentrate on the interesting code, the object graphs, and the relationships between objects in your application. {app_code} {app_code} {app_code} {app_code} {app_code} {app_code} {app_code} {app_code} {app_code} {app_code} Reference counting manually Automatic Reference Counting retain/release code retain/release code retain/release code retain/release code retain/release code retain/release code Time to produce Time to produce Summary ARC works by adding code at compile time to ensure that objects live as long as necessary, but no longer. Conceptually, it follows the same memory management conventions as manual reference counting (described in Advanced Memory Management Programming Guide ) by adding the appropriate memory management calls for you. 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 3 Transitioning to ARC Release NotesIn order for the compiler to generate correct code, ARC restricts the methods you can use and how you use toll-free bridging (see “Toll-Free Bridged Types”). ARC also introduces new lifetime qualifiers for object references and declared properties. ARC is supported in Xcode 4.2 for OS X v10.6 and v10.7 (64-bit applications) and for iOS 4 and iOS 5. Weak references are not supported in OS X v10.6 and iOS 4. Xcode provides a tool that automates the mechanical parts of the ARC conversion (such as removing retain and release calls) and helps you to fix issues the migrator can’t handle automatically (choose Edit > Refactor > Convert to Objective-C ARC). The migration tool converts all filesin a project to use ARC. You can also choose to use ARC on a per-file basis if it’s more convenient for you to use manual reference counting for some files. See also: ● Advanced Memory Management Programming Guide ● Memory Management Programming Guide for Core Foundation ARC Overview Instead of you having to remember when to use retain, release, and autorelease, ARC evaluates the lifetime requirements of your objects and automatically inserts appropriate memory management calls for you at compile time. The compiler also generates appropriate dealloc methods for you. In general, if you’re only using ARC the traditional Cocoa naming conventions are important only if you need to interoperate with code that uses manual reference counting. A complete and correct implementation of a Person class might look like this: @interface Person : NSObject @property NSString *firstName; @property NSString *lastName; @property NSNumber *yearOfBirth; @property Person *spouse; @end @implementation Person @end Transitioning to ARC Release Notes ARC Overview 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 4(Object properties are strong by default; the strong attribute is described in “ARC Introduces New Lifetime Qualifiers” (page 7).) Using ARC, you could implement a contrived method like this: - (void)contrived { Person *aPerson = [[Person alloc] init]; [aPerson setFirstName:@"William"]; [aPerson setLastName:@"Dudney"]; [aPerson setYearOfBirth:[[NSNumber alloc] initWithInteger:2011]]; NSLog(@"aPerson: %@", aPerson); } ARC takes care of memory management so that neither the Person nor the NSNumber objects are leaked. You could also safely implement a takeLastNameFrom: method of Person like this: - (void)takeLastNameFrom:(Person *)person { NSString *oldLastname = [self lastName]; [self setLastName:[person lastName]]; NSLog(@"Lastname changed from %@ to %@", oldLastname, [self lastName]); } ARC ensures that oldLastName is not deallocated before the NSLog statement. ARC Enforces New Rules To work, ARC imposes some new rules that are not present when using other compiler modes. The rules are intended to provide a fully reliable memory management model; in some cases, they simply enforce best practice, in some others they simplify your code or are obvious corollaries of your not having to deal with memory management. If you violate these rules, you get an immediate compile-time error, not a subtle bug that may become apparent at runtime. ● You cannot explicitly invoke dealloc, or implement or invoke retain, release, retainCount, or autorelease. The prohibition extends to using @selector(retain), @selector(release), and so on. Transitioning to ARC Release Notes ARC Overview 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 5You may implement a dealloc method if you need to manage resources other than releasing instance variables. You do not have to (indeed you cannot) release instance variables, but you may need to invoke [systemClassInstance setDelegate:nil] on system classes and other code that isn’t compiled using ARC. Custom dealloc methods in ARC do not require a call to [super dealloc] (it actually results in a compiler error). The chaining to super is automated and enforced by the compiler. You can still use CFRetain, CFRelease, and other related functions with Core Foundation-style objects (see “Managing Toll-Free Bridging” (page 12)). ● You cannot use NSAllocateObject or NSDeallocateObject. You create objects using alloc; the runtime takes care of deallocating objects. ● You cannot use object pointers in C structures. Rather than using a struct, you can create an Objective-C class to manage the data instead. ● There is no casual casting between id and void *. You must use special caststhat tell the compiler about object lifetime. You need to do thisto cast between Objective-C objects and Core Foundation types that you pass as function arguments. For more details, see “Managing Toll-Free Bridging” (page 12). ● You cannot use NSAutoreleasePool objects. ARC provides @autoreleasepool blocks instead. These have an advantage of being more efficient than NSAutoreleasePool. ● You cannot use memory zones. There is no need to use NSZone any more—they are ignored by the modern Objective-C runtime anyway. To allow interoperation with manual retain-release code, ARC imposes a constraint on method naming: ● You cannot give an accessor a name that begins with new. This in turn means that you can’t, for example, declare a property whose name begins with new unless you specify a different getter: // Won't work: @property NSString *newTitle; // Works: @property (getter=theNewTitle) NSString *newTitle; Transitioning to ARC Release Notes ARC Overview 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 6ARC Introduces New Lifetime Qualifiers ARC introduces several new lifetime qualifiers for objects, and weak references. A weak reference does not extend the lifetime of the object it points to, and automatically becomes nil when there are no strong references to the object. You should take advantage of these qualifiers to manage the object graphs in your program. In particular, ARC does not guard against strong reference cycles (previously known as retain cycles—see “Practical Memory Management”). Judicious use of weak relationships will help to ensure you don’t create cycles. Property Attributes The keywords weak and strong are introduced as new declared property attributes, asshown in the following examples. // The following declaration is a synonym for: @property(retain) MyClass *myObject; @property(strong) MyClass *myObject; // The following declaration is similar to "@property(assign) MyClass *myObject;" // except that if the MyClass instance is deallocated, // the property value is set to nil instead of remaining as a dangling pointer. @property(weak) MyClass *myObject; Under ARC, strong is the default for object types. Variable Qualifiers You use the following lifetime qualifiers for variables just like you would, say, const. __strong __weak __unsafe_unretained __autoreleasing ● __strong is the default. An object remains “alive” as long as there is a strong pointer to it. ● __weak specifies a reference that does not keep the referenced object alive. A weak reference is set to nil when there are no strong references to the object. Transitioning to ARC Release Notes ARC Overview 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 7● __unsafe_unretained specifies a reference that does not keep the referenced object alive and is not set to nil when there are no strong references to the object. If the object it references is deallocated, the pointer is left dangling. ● __autoreleasing is used to denote argumentsthat are passed by reference (id *) and are autoreleased on return. You should decorate variables correctly. When using qualifiers in an object variable declaration, the correct format is: ClassName * qualifier variableName; for example: MyClass * __weak myWeakReference; MyClass * __unsafe_unretained myUnsafeReference; Other variants are technically incorrect but are “forgiven” by the compiler. To understand the issue, see http://cdecl.org/. Take care when using __weak variables on the stack. Consider the following example: NSString * __weak string = [[NSString alloc] initWithFormat:@"First Name: %@", [self firstName]]; NSLog(@"string: %@", string); Although string is used after the initial assignment, there is no other strong reference to the string object at the time of assignment; it is therefore immediately deallocated. The log statement shows that string has a null value. (The compiler provides a warning in this situation.) You also need to take care with objects passed by reference. The following code will work: NSError *error; BOOL OK = [myObject performOperationWithError:&error]; if (!OK) { // Report the error. // ... However, the error declaration is implicitly: Transitioning to ARC Release Notes ARC Overview 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 8NSError * __strong e; and the method declaration would typically be: -(BOOL)performOperationWithError:(NSError * __autoreleasing *)error; The compiler therefore rewrites the code: NSError * __strong error; NSError * __autoreleasing tmp = error; BOOL OK = [myObject performOperationWithError:&tmp]; error = tmp; if (!OK) { // Report the error. // ... The mismatch between the local variable declaration (__strong) and the parameter (__autoreleasing) causesthe compiler to create the temporary variable. You can get the original pointer by declaring the parameter id __strong * when you take the address of a __strong variable. Alternatively you can declare the variable as __autoreleasing. Use Lifetime Qualifiers to Avoid Strong Reference Cycles You can use lifetime qualifiers to avoid strong reference cycles. For example, typically if you have a graph of objects arranged in a parent-child hierarchy and parents need to refer to their children and vice versa, then you make the parent-to-child relationship strong and the child-to-parent relationship weak. Other situations may be more subtle, particularly when they involve block objects. In manual reference counting mode, __block id x; hasthe effect of not retaining x. In ARC mode, __block id x; defaults to retaining x (just like all other values). To get the manual reference counting mode behavior under ARC, you could use __unsafe_unretained __block id x;. As the name __unsafe_unretained implies, however, having a non-retained variable is dangerous (because it can dangle) and is therefore discouraged. Two better options are to either use __weak (if you don’t need to support iOS 4 or OS X v10.6), or set the __block value to nil to break the retain cycle. The following code fragment illustrates this issue using a pattern that is sometimes used in manual reference counting. Transitioning to ARC Release Notes ARC Overview 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 9MyViewController *myController = [[MyViewController alloc] init…]; // ... myController.completionHandler = ^(NSInteger result) { [myController dismissViewControllerAnimated:YES completion:nil]; }; [self presentViewController:myController animated:YES completion:^{ [myController release]; }]; As described, instead, you can use a __block qualifier and set the myController variable to nil in the completion handler: MyViewController * __block myController = [[MyViewController alloc] init…]; // ... myController.completionHandler = ^(NSInteger result) { [myController dismissViewControllerAnimated:YES completion:nil]; myController = nil; }; Alternatively, you can use a temporary __weak variable. The following example illustrates a simple implementation: MyViewController *myController = [[MyViewController alloc] init…]; // ... MyViewController * __weak weakMyViewController = myController; myController.completionHandler = ^(NSInteger result) { [weakMyViewController dismissViewControllerAnimated:YES completion:nil]; }; For non-trivial cycles, however, you should use: MyViewController *myController = [[MyViewController alloc] init…]; // ... MyViewController * __weak weakMyController = myController; myController.completionHandler = ^(NSInteger result) { Transitioning to ARC Release Notes ARC Overview 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 10MyViewController *strongMyController = weakMyController; if (strongMyController) { // ... [strongMyController dismissViewControllerAnimated:YES completion:nil]; // ... } else { // Probably nothing... } }; In some cases you can use __unsafe_unretained if the class isn’t __weak compatible. This can, however, become impractical for nontrivial cycles because it can be hard or impossible to validate that the __unsafe_unretained pointer is still valid and still points to the same object in question. ARC Uses a New Statement to Manage Autorelease Pools Using ARC, you cannot manage autorelease pools directly using the NSAutoreleasePool class. Instead, you use @autoreleasepool blocks: @autoreleasepool { // Code, such as a loop that creates a large number of temporary objects. } This simple structure allows the compiler to reason about the reference count state. On entry, an autorelease pool is pushed. On normal exit (break, return, goto, fall-through, and so on) the autorelease pool is popped. For compatibility with existing code, if exit is due to an exception, the autorelease pool is not popped. Thissyntax is available in all Objective-C modes. It is more efficient than using the NSAutoreleasePool class; you are therefore encouraged to adopt it in place of using the NSAutoreleasePool. Patterns for Managing Outlets Become Consistent Across Platforms The patternsfor declaring outletsin iOS and OS X change with ARC and become consistent across both platforms. The pattern you should typically adopt is: outletsshould be weak, except for those from File’s Owner to top-level objects in a nib file (or a storyboard scene) which should be strong. Full details are given in “Nib Files” in Resource Programming Guide . Transitioning to ARC Release Notes ARC Overview 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 11Stack Variables Are Initialized with nil Using ARC,strong, weak, and autoreleasing stack variables are now implicitly initialized with nil. For example: - (void)myMethod { NSString *name; NSLog(@"name: %@", name); } will log null for the value of name rather than perhaps crashing. Use Compiler Flags to Enable and Disable ARC You enable ARC using a new -fobjc-arc compiler flag. You can also choose to use ARC on a per-file basis if it’s more convenient for you to use manual reference counting for some files. For projects that employ ARC as the default approach, you can disable ARC for a specific file using a new -fno-objc-arc compiler flag for that file. ARC is supported in Xcode 4.2 and later OS X v10.6 and later (64-bit applications) and for iOS 4 and later. Weak references are not supported in OS X v10.6 and iOS 4. There is no ARC support in Xcode 4.1 and earlier. Managing Toll-Free Bridging In many Cocoa applications, you need to use Core Foundation-style objects, whether from the Core Foundation framework itself (such as CFArrayRef or CFMutableDictionaryRef) or from frameworks that adopt Core Foundation conventions such as Core Graphics (you might use types like CGColorSpaceRef and CGGradientRef). The compiler does not automatically manage the lifetimes of Core Foundation objects; you must call CFRetain and CFRelease (or the corresponding type-specific variants) as dictated by the Core Foundation memory management rules (see Memory Management Programming Guide for Core Foundation ). If you cast between Objective-C and Core Foundation-style objects, you need to tell the compiler about the ownership semantics of the object using either a cast (defined in objc/runtime.h) or a Core Foundation-style macro (defined in NSObject.h): ● __bridge transfers a pointer between Objective-C and Core Foundation with no transfer of ownership. ● __bridge_retained or CFBridgingRetain casts an Objective-C pointer to a Core Foundation pointer and also transfers ownership to you. Transitioning to ARC Release Notes Managing Toll-Free Bridging 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 12You are responsible for calling CFRelease or a related function to relinquish ownership of the object. ● __bridge_transfer or CFBridgingRelease moves a non-Objective-C pointer to Objective-C and also transfers ownership to ARC. ARC is responsible for relinquishing ownership of the object. For example, if you had code like this: - (void)logFirstNameOfPerson:(ABRecordRef)person { NSString *name = (NSString *)ABRecordCopyValue(person, kABPersonFirstNameProperty); NSLog(@"Person's first name: %@", name); [name release]; } you could replace it with: - (void)logFirstNameOfPerson:(ABRecordRef)person { NSString *name = (NSString *)CFBridgingRelease(ABRecordCopyValue(person, kABPersonFirstNameProperty)); NSLog(@"Person's first name: %@", name); } The Compiler Handles CF Objects Returned From Cocoa Methods The compiler understands Objective-C methods that return Core Foundation types follow the historical Cocoa naming conventions (see Advanced Memory Management Programming Guide ). For example, the compiler knows that, in iOS, the CGColor returned by the CGColor method of UIColor is not owned. You must still use an appropriate type cast, as illustrated by this example: NSMutableArray *colors = [NSMutableArray arrayWithObject:(id)[[UIColor darkGrayColor] CGColor]]; [colors addObject:(id)[[UIColor lightGrayColor] CGColor]]; Transitioning to ARC Release Notes Managing Toll-Free Bridging 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 13Cast Function Parameters Using Ownership Keywords When you cast between Objective-C and Core Foundation objectsin function calls, you need to tell the compiler about the ownership semantics of the passed object. The ownership rules for Core Foundation objects are those specified in the Core Foundation memory management rules (see Memory Management Programming Guide for Core Foundation ); rules for Objective-C objects are specified in Advanced Memory Management Programming Guide . In the following code fragment, the array passed to the CGGradientCreateWithColors function requires an appropriate cast. Ownership of the object returned by arrayWithObjects: is not passed to the function, thus the cast is __bridge. NSArray *colors = <#An array of colors#>; CGGradientRef gradient = CGGradientCreateWithColors(colorSpace, (__bridge CFArrayRef)colors, locations); The code fragment is shown in context in the following method implementation. Notice also the use of Core Foundation memory management functions where dictated by the Core Foundation memory management rules. - (void)drawRect:(CGRect)rect { CGContextRef ctx = UIGraphicsGetCurrentContext(); CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceGray(); CGFloat locations[2] = {0.0, 1.0}; NSMutableArray *colors = [NSMutableArray arrayWithObject:(id)[[UIColor darkGrayColor] CGColor]]; [colors addObject:(id)[[UIColor lightGrayColor] CGColor]]; CGGradientRef gradient = CGGradientCreateWithColors(colorSpace, (__bridge CFArrayRef)colors, locations); CGColorSpaceRelease(colorSpace); // Release owned Core Foundation object. CGPoint startPoint = CGPointMake(0.0, 0.0); CGPoint endPoint = CGPointMake(CGRectGetMaxX(self.bounds), CGRectGetMaxY(self.bounds)); CGContextDrawLinearGradient(ctx, gradient, startPoint, endPoint, kCGGradientDrawsBeforeStartLocation | kCGGradientDrawsAfterEndLocation); CGGradientRelease(gradient); // Release owned Core Foundation object. } Transitioning to ARC Release Notes Managing Toll-Free Bridging 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 14Common Issues While Converting a Project When migrating existing projects, you are likely to run into various issues. Here are some common issues, together with solutions. You can’t invoke retain, release, or autorelease. This is a feature. You also can’t write: while ([x retainCount]) { [x release]; } You can’t invoke dealloc. You typically invoke dealloc if you are implementing a singleton or replacing an object in an init methods. Forsingletons, use the shared instance pattern. In init methods, you don't have to call dealloc anymore, because the object will be freed when you overwrite self. You can’t use NSAutoreleasePool objects. Use the new @autoreleasepool{} construct instead. This forces a block structure on your autorelease pool, and is about six times faster than NSAutoreleasePool. @autoreleasepool even works in non-ARC code. Because @autoreleasepool is so much faster than NSAutoreleasePool, many old “performance hacks” can simply be replaced with unconditional @autoreleasepool. The migrator handles simple uses of NSAutoreleasePool, but it can't handle complex conditional cases, or cases where a variable is defined inside the body of the new @autoreleasepool and used after it. ARC requires you to assign the result of [super init] to self in init methods. The following is invalid in ARC init methods: [super init]; The simple fix is to change it to: self = [super init]; The proper fix is to do that, and check the result for nil before continuing: self = [super init]; if (self) { ... Transitioning to ARC Release Notes Common Issues While Converting a Project 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 15You can’t implement custom retain or release methods. Implementing custom retain or release methods breaks weak pointers. There are several common reasons for wanting to provide custom implementations: ● Performance. Please don’t do this any more; the implementation of retain and release for NSObject is much faster now. If you still find problems, please file bugs. ● To implement a custom weak pointer system. Use __weak instead. ● To implement singleton class. Use the shared instance pattern instead. Alternatively, use class instead of instance methods, which avoids having to allocate the object at all. “Assigned” instance variables become strong. Before ARC, instance variables were non-owning references—directly assigning an object to an instance variable did not extend the lifetime of the object. To make a property strong, you usually implemented or synthesized accessor methods that invoked appropriate memory management methods; in contrast, you may have implemented accessor methods like those shown in the following example to maintain a weak property. @interface MyClass : Superclass { id thing; // Weak reference. } // ... @end @implementation MyClass - (id)thing { return thing; } - (void)setThing:(id)newThing { thing = newThing; } // ... Transitioning to ARC Release Notes Common Issues While Converting a Project 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 16@end With ARC, instance variables are strong references by default—assigning an object to an instance variable directly does extend the lifetime of the object. The migration tool is not able to determine when an instance variable is intended to be weak. To maintain the same behavior as before, you must mark the instance variable as being weak, or use a declared property. @interface MyClass : Superclass { id __weak thing; } // ... @end @implementation MyClass - (id)thing { return thing; } - (void)setThing:(id)newThing { thing = newThing; } // ... @end Or: @interface MyClass : Superclass @property (weak) id thing; // ... @end @implementation MyClass @synthesize thing; // ... @end Transitioning to ARC Release Notes Common Issues While Converting a Project 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 17You can't use strong ids in C structures. For example, the following code won’t compile: struct X { id x; float y; }; This is because x defaults to strongly retained and the compiler can’t safely synthesize all the code required to make it work correctly. For example, if you pass a pointer to one of these structures through some code that ends up doing a free, each id would have to be released before the struct is freed. The compiler cannot reliably do this, so strong ids in structures are disallowed completely in ARC mode. There are a few possible solutions: 1. Use Objective-C objects instead of structs. This is considered to be best practice anyway. 2. If using Objective-C objects is sub-optimal, (maybe you want a dense array of these structs) then consider using a void* instead. This requires the use of the explicit casts, described below. 3. Mark the object reference as __unsafe_unretained. This approach may be useful for the semi-common patterns like this: struct x { NSString *S; int X; } StaticArray[] = { @"foo", 42, @"bar, 97, ... }; You declare the structure as: struct x { NSString * __unsafe_unretained S; int X; } This may be problematic and is unsafe if the object could be released out from under the pointer, but it is very useful for things that are known to be around forever like constant string literals. You can’t directly cast between id and void* (including Core Foundation types). This is discussed in greater detail in “Managing Toll-Free Bridging” (page 12). Transitioning to ARC Release Notes Common Issues While Converting a Project 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 18Frequently Asked Questions How do I think about ARC? Where does it put the retains/releases? Try to stop thinking about where the retain/release calls are put and think about your application algorithms instead. Think about “strong and weak” pointers in your objects, about object ownership, and about possible retain cycles. Do I still need to write dealloc methods for my objects? Maybe. Because ARC does not automate malloc/free, management of the lifetime of Core Foundation objects, file descriptors, and so on, you still free such resources by writing a dealloc method. You do not have to (indeed cannot) release instance variables, but you may need to invoke [self setDelegate:nil] on system classes and other code that isn’t compiled using ARC. dealloc methods in ARC do not require—or allow—a call to [super dealloc]; the chaining to super is handled and enforced by the runtime. Are retain cycles still possible in ARC? Yes. ARC automates retain/release, and inherits the issue of retain cycles. Fortunately, code migrated to ARC rarely starts leaking, because properties already declare whether the properties are retaining or not. How do blocks work in ARC? Blocks “just work” when you pass blocks up the stack in ARC mode, such as in a return. You don’t have to call Block Copy any more. You still need to use [^{} copy] when passing “down” the stack into arrayWithObjects: and other methods that do a retain. The one thing to be aware of isthat NSString * __block myString isretained in ARC mode, not a possibly dangling pointer. To get the previous behavior, use __block NSString * __unsafe_unretained myString or (better still) use __block NSString * __weak myString. Can I develop applications for OS X with ARC using Snow Leopard? No. The Snow Leopard version of Xcode 4.2 doesn’t support ARC at all on OS X, because it doesn’t include the 10.7 SDK. Xcode 4.2 for Snow Leopard does support ARC for iOS though, and Xcode 4.2 for Lion supports both OS X and iOS. This means you need a Lion system to build an ARC application that runs on Snow Leopard. Can I create a C array of retained pointers under ARC? Transitioning to ARC Release Notes Frequently Asked Questions 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 19Yes, you can, as illustrated by this example: // Note calloc() to get zero-filled memory. __strong SomeClass **dynamicArray = (__strong SomeClass **)calloc(sizeof(SomeClass *), entries); for (int i = 0; i < entries; i++) { dynamicArray[i] = [[SomeClass alloc] init]; } // When you're done, set each entry to nil to tell ARC to release the object. for (int i = 0; i < entries; i++) { dynamicArray[i] = nil; } free(dynamicArray); There are a number of aspects to note: ● You will need to write __strong SomeClass ** in some cases, because the default is __autoreleasing SomeClass **. ● The allocated memory must be zero-filled. ● You must set each element to nil before freeing the array (memset or bzero will not work). ● You should avoid memcpy or realloc. Is ARC slow? It depends on what you’re measuring, but generally “no.” The compiler efficiently eliminates many extraneous retain/release calls and much effort has been invested in speeding up the Objective-C runtime in general. In particular, the common “return a retain/autoreleased object” pattern is much faster and does not actually put the object into the autorelease pool, when the caller of the method is ARC code. One issue to be aware of is that the optimizer is not run in common debug configurations, so expect to see a lot more retain/release traffic at -O0 than at -Os. Does ARC work in ObjC++ mode? Yes. You can even put strong/weak ids in classes and containers. The ARC compiler synthesizes retain/release logic in copy constructors and destructors etc to make this work. Which classes don’t support weak references? Transitioning to ARC Release Notes Frequently Asked Questions 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 20You cannot currently create weak references to instances of the following classes: NSATSTypesetter, NSColorSpace, NSFont, NSMenuView, NSParagraphStyle, NSSimpleHorizontalTypesetter, and NSTextView. Note: In addition, in OS X v10.7, you cannot create weak referencesto instances of NSFontManager, NSFontPanel, NSImage, NSTableCellView, NSViewController, NSWindow, and NSWindowController. In addition, in OS X v10.7 no classes in the AV Foundation framework support weak references. For declared properties, you should use assign instead of weak; for variables you should use __unsafe_unretained instead of __weak. In addition, you cannot create weak references from instances of NSHashTable, NSMapTable, or NSPointerArray under ARC. What do I have to do when subclassing NSCell or another class that uses NSCopyObject? Nothing special. ARC takes care of cases where you had to previously add extra retains explicitly. With ARC, all copy methods should just copy over the instance variables. Can I opt out of ARC for specific files? Yes. Transitioning to ARC Release Notes Frequently Asked Questions 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 21When you migrate a project to use ARC, the -fobjc-arc compiler flag is set as the default for all Objective-C source files. You can disable ARC for a specific class using the -fno-objc-arc compiler flag for that class. In Xcode, in the target Build Phasestab, open the Compile Sources group to reveal the source file list. Double-click the file for which you want to set the flag, enter -fno-objc-arc in the pop-up panel, then click Done. Is GC (Garbage Collection) deprecated on the Mac? Garbage collection is deprecated in OS X Mountain Lion v10.8, and will be removed in a future version of OS X. Automatic Reference Counting is the recommended replacement technology. To aid in migrating existing applications, the ARC migration tool in Xcode 4.3 and later supports migration of garbage collected OS X applications to ARC. Note: For apps targeting the Mac App Store, Apple strongly recommends you replace garbage collection with ARC as soon as feasible, because Mac App Store guidelines (see App Store Review Guidelines for Mac Apps) prohibit the use of deprecated technologies. Transitioning to ARC Release Notes Frequently Asked Questions 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 22This table describes the changes to Transitioning to ARC Release Notes. Date Notes 2012-07-17 Updated for OS X v10.8. 2012-03-14 Noted that under ARC properties are strong by default. 2012-02-16 Corrected out-of-date advice regarding C++ integration. 2012-01-09 Added note to search for weak references. First version of a document that describes how to transition code from manual retain/release to use ARC. 2011-10-12 2012-07-17 | © 2012 Apple Inc. All Rights Reserved. 23 Document Revision HistoryApple Inc. © 2012 Apple Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrievalsystem, or transmitted, in any form or by any means, mechanical, electronic, photocopying, recording, or otherwise, without prior written permission of Apple Inc., with the following exceptions: Any person is hereby authorized to store documentation on a single computer for personal use only and to print copies of documentation for personal use provided that the documentation contains Apple’s copyright notice. No licenses, express or implied, are granted with respect to any of the technology described in this document. Apple retains all intellectual property rights associated with the technology described in this document. This document is intended to assist application developers to develop applications only for Apple-labeled computers. Apple Inc. 1 Infinite Loop Cupertino, CA 95014 408-996-1010 Apple, the Apple logo, Cocoa, Leopard, Mac, Objective-C, OS X, Snow Leopard, and Xcode are trademarks of Apple Inc., registered in the U.S. and other countries. App Store and Mac App Store are service marks of Apple Inc. iOS is a trademark or registered trademark of Cisco in the U.S. and other countries and is used under license. Even though Apple has reviewed this document, APPLE MAKES NO WARRANTY OR REPRESENTATION, EITHER EXPRESS OR IMPLIED, WITH RESPECT TO THIS DOCUMENT, ITS QUALITY, ACCURACY, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.ASARESULT, THISDOCUMENT IS PROVIDED “AS IS,” AND YOU, THE READER, ARE ASSUMING THE ENTIRE RISK AS TO ITS QUALITY AND ACCURACY. IN NO EVENT WILL APPLE BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL,OR CONSEQUENTIAL DAMAGES RESULTING FROM ANY DEFECT OR INACCURACY IN THIS DOCUMENT, even if advised of the possibility of such damages. THE WARRANTY AND REMEDIES SET FORTH ABOVE ARE EXCLUSIVE AND IN LIEU OF ALL OTHERS, ORAL OR WRITTEN, EXPRESS OR IMPLIED. No Apple dealer, agent, or employee is authorized to make any modification, extension, or addition to this warranty. Some states do not allow the exclusion or limitation of implied warranties or liability for incidental or consequential damages, so the above limitation or exclusion may not apply to you. This warranty gives you specific legal rights, and you may also have other rights which vary from state to state. App Sandbox Design GuideContents About App Sandbox 5 At a Glance 5 How to Use This Document 6 Prerequisites 6 See Also 6 App Sandbox Quick Start 8 Create the Xcode Project 8 Enable App Sandbox 10 Create a Code Signing Certificate for Testing 10 Specify the Code Signing Identity 11 Confirm That the App Is Sandboxed 12 Resolve an App Sandbox Violation 13 App Sandbox in Depth 15 The Need for a Last Line of Defense 15 Container Directories and File System Access 16 The App Sandbox Container Directory 16 The Application Group Container Directory 17 Powerbox and File System Access Outside of Your Container 17 Open and Save Dialog Behavior with App Sandbox 19 Entitlements and System Resource Access 20 Security-Scoped Bookmarks and Persistent Resource Access 21 Two Distinct Types of Security-Scoped Bookmark 21 Using Security-Scoped Bookmarks 22 App Sandbox and Code Signing 24 External Tools, XPC Services, and Privilege Separation 26 Designing for App Sandbox 27 Six Steps for Adopting App Sandbox 27 Determine Whether Your App Is Suitable for Sandboxing 27 Design a Development and Distribution Strategy 29 Resolve API Incompatibilities 29 Opening, Saving, and Tracking Documents 29 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 2Retaining Access to File System Resources 29 Creating a Login Item for Your App 30 Accessing User Data 30 Accessing Preferences of Other Apps 30 Apply the App Sandbox Entitlements You Need 31 Add Privilege Separation Using XPC 32 Implement a Migration Strategy 32 Migrating an App to a Sandbox 33 Creating a Container Migration Manifest 34 Undoing a Migration for Testing 36 An Example Container Migration Manifest 36 Use Variables to Specify Support-File Directories 37 Document Revision History 39 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 3 ContentsTables and Listings App Sandbox in Depth 15 Table 2-1 The App Sandbox mindset 15 Table 2-2 Open and Save class inheritance with App Sandbox 20 Migrating an App to a Sandbox 33 Table 4-1 How system directory variables resolve depending on context 37 Table 4-2 Variables for support-file directories 38 Listing 4-1 An example container migration manifest 36 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 4App Sandbox provides a last line of defense against stolen, corrupted, or deleted user data if malicious code exploits your app. App Sandbox also minimizes the damage from coding errors in your app or in frameworks you link against. Your app All system resources All user data Unrestricted access Other system resources Other user data Your app Unrestricted access No access Without App Sandbox With App Sandbox Your sandbox App Sandbox is an access control technology provided in OS X, enforced at the kernel level. Its strategy is twofold: 1. App Sandbox enables you to describe how your app interacts with the system. The system then grants your app the access it needs to get its job done, and no more. 2. App Sandbox allows the user to transparently grant your app additional access by way of Open and Save dialogs, drag and drop, and other familiar user interactions. At a Glance Based on simple security principles, App Sandbox provides strong defense against damage from malicious code. The elements of App Sandbox are container directories, entitlements, user-determined permissions, privilege separation, and kernel enforcement. It’s up to you to understand these elements and then to use your understanding to create a plan for adopting App Sandbox. 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 5 About App SandboxRelevant chapters: “App Sandbox Quick Start” (page 8), “App Sandbox in Depth” (page 15) After you understand the basics, look at your app in light of this security technology. First, determine if your app issuitable forsandboxing. Most apps are. Design your developmentstrategy, resolve API incompatibilities, determine which entitlements you need, and consider applying privilege separation to maximize the defensive value of App Sandbox. Relevant chapter: “Designing for App Sandbox” (page 27) Some file system locations that your app uses are different when you adopt App Sandbox. In particular, you gain a container directory to be used for app support files, databases, caches, and other files apart from user documents. OS X and Xcode support migration of files from their legacy locations to your container. Relevant chapter: “Migrating an App to a Sandbox” (page 33) How to Use This Document To get up and running with App Sandbox, perform the tutorial in “App Sandbox Quick Start” (page 8). Before sandboxing an app you intend to distribute, be sure you understand “App Sandbox in Depth” (page 15). When you’re ready to startsandboxing a new app, or to convert an existing app to adopt App Sandbox, read “Designing for App Sandbox” (page 27). If you’re providing a new, sandboxed version of your app to users already running a version that is not sandboxed, read “Migrating an App to a Sandbox” (page 33). Prerequisites Before you read this document, make sure you understand the place of App Sandbox and code signing in the overall OS X development process by reading Mac App Programming Guide . See Also To complement the damage containment provided by App Sandbox, you must provide a first line of defense by adopting secure coding practices throughout your app. To learn how, read Security Overview and Secure Coding Guide . About App Sandbox How to Use This Document 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 6An important step in adopting App Sandbox is requesting entitlements for your app. For details on all the available entitlements, see Entitlement Key Reference . You can enhance the benefits of App Sandbox in a full-featured app by implementing privilege separation. You do this using XPC, an OS X implementation of interprocess communication. To learn the details of using XPC, read Daemons and Services Programming Guide . About App Sandbox See Also 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 7In this Quick Start you get an OS X app up and running in a sandbox. You verify that the app isindeed sandboxed and then learn how to troubleshoot and resolve a typical App Sandbox error. The apps you use are Xcode, Keychain Access, Activity Monitor, and Console. Create the Xcode Project The app you create in this Quick Start uses a WebKit web view and consequently uses a network connection. Under App Sandbox, network connections don’t work unless you specifically allow them—making this a good example app for learning about sandboxing. To create the Xcode project for this Quick Start 1. In Xcode 4, create a new Xcode project for an OS X Cocoa application. ● Name the project AppSandboxQuickStart. ● Set a company identifier, such as com.yourcompany, if none is already set. ● Ensure that Use Automatic Reference Counting is selected and that the other checkboxes are unselected. 2. In the project navigator, click the MainMenu nib file. The Interface Builder canvas appears. 3. In the Xcode dock, click the Window object. The app’s window is now visible on the canvas. 4. In the object library (in the utilities area), locate the WebView object. 5. Drag a web view onto the window on the canvas. 6. (Optional) To improve the display of the web view in the running app, perform the following steps: ● Drag the sizing controls on the web view so that it completely fills the window’s main view. ● Using the size inspector for the web view, ensure that all of the inner and outer autosizing contraints are active. 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 8 App Sandbox Quick Start7. Create and connect an outlet for the web view in the AppDelegate class. In Xcode, use the following specification: Outlet connection source The WebView object of the MainMenu nib file. Outlet variable location The interface block of the AppDelegate.h header file. Outlet name webView Storage weak At this point, if you were to build the app, Xcode would report an error because the project doesn’t yet use WebKit but does have a web view in the nib file. You take care of this in the next step. 8. Add the WebKit framework to the app. ● Import the WebKit framework by adding the following statement above the interface block in the AppDelegate.h header file: #import ● Link the WebKit framework to the Quick Start project as a required framework. 9. Add the following awakeFromNib method to the AppDelegate.m implementation file: - (void) awakeFromNib { [self.webView.mainFrame loadRequest: [NSURLRequest requestWithURL: [NSURL URLWithString: @"http://www.apple.com"]]]; } On application launch, this method requeststhe specified URL from the computer’s network connection and then sends the result to the web view for display. Now, build and run the app—which is not yet sandboxed and so has free access to system resources including its network sockets. Confirm that the app’s window displays the page you specified in the awakeFromNib method. When done, quit the app. App Sandbox Quick Start Create the Xcode Project 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 9Enable App Sandbox You enable App Sandbox by selecting a checkbox in the Xcode target editor. In Xcode, click the project file in the project navigator and click the AppSandboxQuickStart target, if they’re not already selected. View the Summary tab of the target editor. To enable App Sandbox for the project 1. In the Summary tab of the target editor, click Enable Entitlements. An entitlement is a key-value pair, defined in a property list file, that confers a specific capability or security permission to a target. When you click Enable Entitlements, Xcode automatically checks the Code Sign Application checkbox and the Enable App Sandboxing checkbox. Together, these are the essential projectsettingsfor enabling App Sandbox. When you click Enable Entitlements, Xcode also creates a .entitlements property list file, visible in the project navigator. As you use the graphical entitlementsinterface in the target editor, Xcode updates the property list file. 2. Clear the contents of the iCloud entitlement fields. This Quick Start doesn’t use iCloud. Because Xcode automatically adds iCloud entitlement values when you enable entitlements, delete them as follows: ● In the Summary tab of the target editor,select and then delete the content of the iCloud Key-Value Store field. ● Click the top row in the iCloud Containers field and click the minus button. At this point in the Quick Start, you have enabled App Sandbox but have not yet provided a code signing identity for the Xcode project. Consequently, if you attempt to build the project now, the build fails. You take care of this in the next two sections. Create a Code Signing Certificate for Testing To build a sandboxed app in Xcode, you must have a code signing certificate and its associated private key in your keychain, and then use that certificate’s code signing identity in the project. The entitlements you specify, including the entitlement that enables App Sandbox, become part of the app’s code signature when you build the project. App Sandbox Quick Start Enable App Sandbox 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 10In this section, you create a code signing certificate. This simplified process lets you stay focused on the steps for enabling a sandbox. Important: A code signing certificate that you create as described in this Quick Start is not appropriate to use with an app you intend to distribute. Before you work on sandboxing an app you plan to distribute, read “App Sandbox and Code Signing” (page 24). To create a code signing certificate for testing App Sandbox 1. In Keychain Access (available in Applications/Utilities), choose KeyChain Access > Certificate Assistant > Create a Certificate. Certificate Assistant opens. Note: Before you invoke the “Create a Certificate” menu command, ensure that no key is selected in the Keychain Access main window. If a key is selected, the menu command is not available. 2. In Certificate Assistant, name the certificate something like My Test Certificate. 3. Complete the configuration of the certificate as follows: Identity type Self Signed Root Certificate type Code Signing Let me override defaults unchecked 4. Click Create. 5. In the alert that appears, click Continue. 6. In the Conclusion window, click Done. Your new code signing certificate, and its associated public and private keys, are now available in Keychain Access. Specify the Code Signing Identity Now, configure the Xcode project to use the code signing identity from the certificate you created in the previous task. App Sandbox Quick Start Specify the Code Signing Identity 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 11To specify the code signing identity for the project 1. View the Build Settings tab in the project editor. Take care that you are using the project editor, not the target editor. 2. In the Code Signing section, locate the Code Signing Identity row. 3. Click the value area of the Code Signing Identity row. 4. In the popup menu that opens, choose Other. 5. In the text entry window that opens, enter the exact name of the newly created code signing certificate, then press . If you’re using the suggested name from thisQuick Start, the name you enter is My Test Certificate. Now, build the app. The codesign tool may display an alert asking for permission to use the new certificate. If you do see this alert, click Always Allow. Confirm That the App Is Sandboxed Build and run the Quick Start app. The window opens, but if the app issuccessfully sandboxed, no web content appears. This is because you have not yet conferred permission to access a network connection. Apart from blocked behavior, there are two specific signs that an OS X app is successfully sandboxed. To confirm that the Quick Start app is successfully sandboxed 1. In Finder, look at the contents of the ~/Library/Containers/ folder. If the Quick Start app is sandboxed, there is now a container folder named after your app. The name includes the company identifier for the project, so the complete folder name would be, for example, com.yourcompany.AppSandboxQuickStart. The system creates an app’s container folder, for a given user, the first time the user runs the app. 2. In Activity Monitor, check that the system recognizes the app as sandboxed. ● Launch Activity Monitor (available in /Applications/Utilities). ● In Activity Monitor, choose View > Columns. Ensure that the Sandbox menu item is checked. ● In the Sandbox column, confirm that the value for the Quick Start app is Yes. App Sandbox Quick Start Confirm That the App Is Sandboxed 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 12To make it easier to locate the app in Activity monitor, enter the name of the Quick Start app in the Filter field. Tip: If the app crashes when you attempt to run it,specifically by receiving an EXC_BAD_INSTRUCTION signal, the most likely reason is that you previously ran a sandboxed app with the same bundle identifier but a different code signature. This crashing upon launch is an App Sandbox security feature that prevents one app from masquerading as another and thereby gaining access to the other app’s container. You learn how to design and build your apps, in light of this security feature, in “App Sandbox and Code Signing” (page 24). Resolve an App Sandbox Violation An App Sandbox violation occurs if your app tries to do something that App Sandbox does not allow. For example, you have already seen in this Quick Start that the sandboxed app is unable to retrieve content from the web. Fine-grained restriction over access to system resources is the heart of how App Sandbox provides protection should an app become compromised by malicious code. The most common source of App Sandbox violations is a mismatch between the entitlement settings you specified in Xcode and the needs of your app. In this section you observe and then correct an App Sandbox violation. To diagnose an App Sandbox violation 1. Build and run the Quick Start app. The app starts normally, but fails to display the webpage specified in its awakeFromNib method (as you’ve previously observed in “Confirm That the App Is Sandboxed” (page 12)). Because displaying the webpage worked correctly before you sandboxed the app, it is appropriate in this case to suspect an App Sandbox violation. 2. Open Console (available in /Applications/Utilities/) and ensure that All Messages is selected in the sidebar. In the filter field of the Console window, enter sandboxd to display only App Sandbox violations. sandboxd is the name of the App Sandbox daemon that reports on sandbox violations. The relevant messages, as displayed in Console, look similar to the following: 3:56:16 pm sandboxd: ([4928]) AppSandboxQuickS(4928) deny network-outbound 111.30.222.15:80 3:56:16 pm sandboxd: ([4928]) AppSandboxQuickS(4928) deny system-socket App Sandbox Quick Start Resolve an App Sandbox Violation 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 13The problem that generates these console messages is that the Quick Start app does not yet have the entitlement for outbound network access. Tip: To see the full backtraces for either violation, click the paperclip icon near the right edge of the corresponding Console message. The steps in the previous task illustrate the general pattern to use for identifying App Sandbox violations: 1. Confirm that the violation occurs only with App Sandbox enabled in your project. 2. Provoke the violation (such as by attempting to use a network connection, if your app is designed to do that). 3. Look in Console for sandboxd messages. There is also a simple, general pattern to use for resolving such violations. To resolve the App Sandbox violation by adding the appropriate entitlement 1. Quit the Quick Start app. 2. In the Summary tab of the target editor, look for the entitlement that corresponds to the reported sandboxd violation. In this case, the primary error is deny network-outbound. The corresponding entitlement is Allow Outgoing Network Connections. 3. In the Summary tab of the target editor, select the Allow Outgoing Network Connections checkbox. Doing so applies a TRUE value, for the needed entitlement, to the Xcode project. 4. Build and run the app. The intended webpage now displays in the app. In addition, there are no new App Sandbox violation messages in Console. App Sandbox Quick Start Resolve an App Sandbox Violation 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 14The access control mechanisms used by App Sandbox to protect user data are small in number and easy to understand. But the specific steps for you to take, as you adopt App Sandbox, are unique to your app. To determine what those steps are, you must understand the key concepts for this technology. The Need for a Last Line of Defense You secure your app against attack from malware by following the practices recommended in Secure Coding Guide . But despite your best efforts to build an invulnerable barrier—by avoiding buffer overflows and other memory corruptions, preventing exposure of user data, and eliminating other vulnerabilities—your app can be exploited by malicious code. An attacker needs only to find a single hole in your defenses, or in any of the frameworks and libraries that you link against, to gain control of your app’s interactions with the system. App Sandbox is designed to confront this scenario head on by letting you describe your app’s intended interactions with the system. The system then grants your app only the access your app needs to get its job done. If malicious code gains control of a properly sandboxed app, it is left with access to only the files and resources in the app’s sandbox. To successfully adopt App Sandbox, use a different mindset than you might be accustomed to, as suggested in Table 2-1. Table 2-1 The App Sandbox mindset When developing… When adopting App Sandbox… Add features Minimize system resource use Take advantage of access throughout your app Partition functionality, then distrust each part Use the most convenient API Use the most secure API View restrictions as limitations View restrictions as safeguards 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 15 App Sandbox in DepthWhen designing for App Sandbox, you are planning for the following worst-case scenario: Despite your best efforts, malicious code breaches an unintended security hole—either in your code or in a framework you’ve linked against. Capabilities you’ve added to your app become capabilities of the hostile code. Keep this in mind as you read the rest of this document. Container Directories and File System Access When you adopt App Sandbox, the system provides a special directory for use by your app—and only by your app—called a container. Each user on a system gets an individual container for your app, within their home directory; your app has unfettered read/write access to the container for the current user. The App Sandbox Container Directory The container has the following characteristics: ● It is located at a system-defined path, within the user’s home directory, that you can obtain by calling the NSHomeDirectory function. ● Your app has unrestricted read/write access to the container and its subdirectories. ● OS X path-finding APIs (above the POSIX layer) refer to locations that are specific to your app. Most of these path-finding APIsrefer to locationsrelative to your app’s container. For example, the container includes an individual Library directory (specified by the NSLibraryDirectory search path constant) for use only by your app, with individual Application Support and Preferences subdirectories. Using your container forsupport filesrequires no code change (from the pre-sandbox version of your app) but may require one-time migration, as explained in “Migrating an App to a Sandbox” (page 33). Some path-finding APIs (above the POSIX layer) refer to app-specific locations outside of the user’s home directory. In a sandboxed app, for example, the NSTemporaryDirectory function provides a path to a directory that is outside of the user’s home directory but specific to your app and within your sandbox; you have unrestricted read/write access to it for the current user. The behavior of these path-finding APIs is suitably adjusted for App Sandbox and no code change is needed. ● OS X establishes and enforces the connection between your app and its container by way of your app’s code signature. ● The container isin a hidden location, and so users do not interact with it directly. Specifically, the container is not for user documents. It is for files that your app uses, along with databases, caches, and other app-specific data. For a shoebox-style app, in which you provide the only user interface to the user’s content, that content goes in the container and your app has full access to it. App Sandbox in Depth Container Directories and File System Access 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 16iOS Note: Because it is not for user documents, an OS X container differs from an iOS container—which, in iOS, is the one and only location for user documents. In addition, an iOS container contains the app itself. This is not so in OS X. iCloud Note: Apple’s iCloud technology, as described in “iCloud Storage”, uses the name “container” as well. There is no functional connection between an iCloud container and an App Sandbox container. Thanks to code signing, no other sandboxed app can gain access to your container, even if it attempts to masquerade as your app by using your bundle identifier. Future versions of your app, however—provided that you use the same code signature and bundle identifier—do reuse your app’s container. The time at which a container directory is created for an App Sandbox–enabled app is when the app is first run. Because a container is within a user’s home folder, each user on a system gets their own container for your app. A given user’s container is created when that user first runs your app. The Application Group Container Directory In addition to per-app containers, beginning in OS X v10.7.4, an application can use entitlements to request access to a shared container that is common to multiple applications produced by the same development team. This container is intended for content that is not user-facing, such as shared caches or databases. Applicationsthat are members of an application group also gain the ability to share Mach and POSIX semaphores and to use certain other IPC mechanisms in conjunction with other group members. These group containers are automatically created or added into each app’s sandbox container as determined by the existence of these keys, and are stored in ~/Library/Group Containers/, where can be whatever name you choose. Your app can obtain the path to the group containers by calling the containerURLForSecurityApplicationGroupIdentifier: method of NSURL. For more details, see “Adding an Application to an Application Group” in Entitlement Key Reference . Powerbox and File System Access Outside of Your Container Your sandboxed app can access file system locations outside of its container in the following three ways: ● At the specific direction of the user App Sandbox in Depth Container Directories and File System Access 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 17● By using entitlements for specific file-system locations (described in “Entitlements and System Resource Access” (page 20)) ● When the file system location is in certain directories that are world readable The OS X security technology that interacts with the user to expand yoursandbox is called Powerbox. Powerbox has no API. Your app uses Powerbox transparently when you use the NSOpenPanel and NSSavePanel classes. You enable Powerbox by setting an entitlement using Xcode, as described in “Enabling User-Selected File Access” in Entitlement Key Reference . When you invoke an Open or Save dialog from your sandboxed app, the window that appears is presented not by AppKit but by Powerbox. Using Powerbox is automatic when you adopt App Sandbox—it requires no code change from the pre-sandbox version of your app. Accessory panelsthat you’ve implemented for opening or saving are faithfully rendered and used. Note: When you adopt App Sandbox, there are some important behavioral differences for the NSOpenPanel and NSSavePanel classes, described in “Open and Save Dialog Behavior with App Sandbox” (page 19). The security benefit provided by Powerbox is that it cannot be manipulated programmatically—specifically, there is no mechanism for hostile code to use Powerbox for accessing the file system. Only a user, by interacting with Open and Save dialogs via Powerbox, can use those dialogs to reach portions of the file system outside of your previously established sandbox. For example, if a user saves a new document, Powerbox expands your sandbox to give your app read/write access to the document. When a user of your app specifies they want to use a file or a folder, the system adds the associated path to your app’s sandbox. Say, for example, a user drags the ~/Documents folder onto your app’s Dock tile (or onto your app’s Finder icon, or into an open window of your app), thereby indicating they want to use that folder. In response, the system makes the ~/Documents folder, its contents, and its subfolders available to your app. If a user instead opens a specific file, or saves to a new file, the system makes the specified file, and that file alone, available to your app. In addition, the system automatically permits a sandboxed app to: ● Connect to system input methods ● Invoke services chosen by the user from the Services menu (only those services flagged as “safe” by the service provider are available to a sandboxed app) ● Open files chosen by the user from the Open Recent menu ● Participate with other apps by way of user-invoked copy and paste App Sandbox in Depth Container Directories and File System Access 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 18● Read files that are world readable, in certain directories, including the following directories: ● /bin ● /sbin ● /usr/bin ● /usr/lib ● /usr/sbin ● /usr/share ● /System After a user hasspecified a file they want to use, that file is within your app’ssandbox. The file isthen vulnerable to attack if your app is exploited by malicious code: App Sandbox provides no protection. To provide protection for the files within your sandbox, follow the recommendations in Secure Coding Guide . A critical aspect of following user intent is that throughout OS X, simulation or alteration of user input is not allowed. This has implications for assistive apps, as described in “Determine Whether Your App Is Suitable for Sandboxing” (page 27). By default, files opened or saved by the user remain within your sandbox until your app terminates, except for files that were open at the time that your app terminates. Such files reopen automatically by way of the OS X Resume feature the next time your app launches, and are automatically added back to your app’ssandbox. To provide persistent access to resources located outside of your container, in a way that doesn’t depend on Resume, use security-scoped bookmarks as explained in “Security-Scoped Bookmarks and Persistent Resource Access” (page 21). Open and Save Dialog Behavior with App Sandbox Certain NSOpenPanel and NSSavePanel methods behave differently when App Sandbox is enabled for your app: ● You cannot invoke the OK button using the ok: method. ● You cannot rewrite the user’sselection using the panel:userEnteredFilename:confirmed: method from the NSOpenSavePanelDelegate protocol. In addition, the effective, runtime inheritance path for the NSOpenPanel and NSSavePanel classesis different with App Sandbox, as illustrated in Table 2-2. App Sandbox in Depth Container Directories and File System Access 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 19Table 2-2 Open and Save class inheritance with App Sandbox Without App Sandbox NSOpenPanel : NSSavePanel : NSPanel : NSWindow : NSResponder : NSObject With App Sandbox NSOpenPanel : NSSavePanel : NSObject Because of this runtime difference, an NSOpenPanel or NSSavePanel object inherits fewer methods with App Sandbox. If you attempt to send a message to an NSOpenPanel or NSSavePanel object, and that method is defined in the NSPanel, NSWindow, or NSResponder classes, the system raises an exception. The Xcode compiler does not issue a warning or error to alert you to this runtime behavior. Entitlements and System Resource Access An app that is notsandboxed has accessto all user-accessible system resources—including the built-in camera and microphone, network sockets, printing, and most of the file system. If successfully attacked by malicious code, such an app can behave as a hostile agent with wide-ranging potential to inflict harm. When you enable App Sandbox for your app, you remove all but a minimalset of privileges and then deliberately restore them, one-by-one, using entitlements. An entitlement is a key-value pair that identifies a specific capability, such as the capability to open an outbound network socket. One special entitlement—Enable App Sandboxing—turns on App Sandbox. When you enable sandboxing, Xcode creates a .entitlements property list file and shows it in the project navigator. If your app requires a capability, request it by adding the corresponding entitlement to your Xcode project using the Summary tab of the target editor. If you don’t require a capability, take care to not include the corresponding entitlement. You request entitlements on a target-by-target basis. If your app has a single target—the main application—you request entitlements only forthat target. If you design your app to use a main application along with helpers (in the form of XPC services), you request entitlements individually, and as appropriate, for each target. You learn more about this in “XPC and Privilege Separation” (page 26). You may require finer-grained control over your app’s entitlements than is available in the Xcode target editor. For example, you might request a temporary exception entitlement because App Sandbox does not support a capability your app needs, such as the ability to send Apple events. To work with temporary exception entitlements, use the Xcode property list editor to edit a target’s .entitlements property list file directly. App Sandbox in Depth Entitlements and System Resource Access 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 20Note: If you request a temporary exception entitlement, be sure to follow the guidance regarding entitlements provided on the iTunes Connect website. In particular, use the Review Notes field in iTunes Connect to explain why your app needs the temporary exception. OS X App Sandbox entitlements are described in “Enabling App Sandbox” in Entitlement Key Reference . For a walk-through of requesting an entitlement for a target in an Xcode project,see “App Sandbox Quick Start” (page 8). Security-Scoped Bookmarks and Persistent Resource Access Your app’s access to file-system locations outside of its container—as granted to your app by way of user intent, such as through Powerbox—does not automatically persist across app launches or system restarts. When your app reopens, you have to start over. (The one exception to this is for files open at the time that your app terminates, which remain in your sandbox thanks to the OS X Resume feature). Starting in OS X v10.7.3, you can retain access to file-system resources by employing a security mechanism, known as security-scoped bookmarks, that preserves user intent. Here are a few examples of app features that can benefit from this: ● A user-selected download, processing, or output folder ● An image browser library file, which points to user-specified images at arbitrary locations ● A complex document format that supports embedded media stored in other locations Two Distinct Types of Security-Scoped Bookmark Security-scoped bookmarks, available starting in OS X v10.7.3, support two distinct use cases: ● An app-scoped bookmark provides your sandboxed app with persistent access to a user-specified file or folder. For example, if your app employs a download or processing folder that is outside of the app container, obtain initial access by presenting an NSOpenPanel dialog to obtain the user’s intent to use a specific folder. Then, create an app-scoped bookmark for that folder and store it as part of the app’s configuration (perhaps in a property list file or using the NSUserDefaults class). With the app-scoped bookmark, your app can obtain future access to the folder. ● A document-scoped bookmark provides a specific document with persistent access to a file. App Sandbox in Depth Security-Scoped Bookmarks and Persistent Resource Access 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 21For example, a code editor typically supports the notion of a project document that refers to other files and needs persistent access to those files. Other examples are an image browser or editor that maintains an image library, in which the library file needs persistent accessto the imagesit owns; or a word processor that supports embedded images, multimedia, or font files in its document format. In these cases, you configure the document format (of the project file, library file, word processing document, and so on) to be able to store security-scoped bookmarks to the files a document refers to. Obtain initial access to a referred item by asking for user intent to use that item. Then, create a document-scoped bookmark for the item and store the bookmark as part of the document’s data. A document-scoped bookmark can be resolved by any app that has access to the bookmark data itself and to the document that owns the bookmark. This supports portability, allowing a user, for example, to send a document to another user; the document’s secure bookmarks remain usable for the recipient. The document can be a flat file or a document distributed as a bundle. A document-scoped bookmark can point only to a file, not a folder, and only to a file that is not in a location used by the system (such as /private or /Library). Using Security-Scoped Bookmarks To use either type of security-scoped bookmark requires you to perform five steps: 1. Set the appropriate entitlement in the target that needs to use security-scoped bookmarks. Do this once per target as part of configuring your Xcode project. 2. Create a security-scoped bookmark. Do this when a user has indicated intent (such as via Powerbox) to use a file-system resource outside of your app’s container, and you want to preserve your app’s ability to access the resource. 3. Resolve the security-scoped bookmark. Do this when your app later (for example, after app relaunch) needs access to a resource you bookmarked in step 2. The result of this step is a security-scoped URL. 4. Explicitly indicate that you want to use the file-system resource whose URL you obtained in step 3. Do this immediately after obtaining the security-scoped URL (or, when you later want to regain access to the resource after having relinquished your access to it). 5. When done using the resource, explicitly indicate that you want to stop using it. Do this as soon as you know that you no longer need access to the resource (typically, after you close it). After you relinquish access to a file-system resource, to use that resource again you must return to step 4 (to again indicate you want to use the resource). If your app is relaunched, you must return to step 3 (to resolve the security-scoped bookmark). App Sandbox in Depth Security-Scoped Bookmarks and Persistent Resource Access 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 22The first step in the preceding list, requesting entitlements, is the prerequisite for using either type of security-scoped bookmark. Perform this step as follows: ● To use app-scoped bookmarksin a target,setthe com.apple.security.files.bookmarks.app-scope entitlement value to true. ● To use document-scoped bookmarks in a target, set the com.apple.security.files.bookmarks.document-scope entitlement value to true. You can request either or both of these entitlements in a target, as needed. These entitlements are available starting in OS X v10.7.3 and are described in “Enabling Security-Scoped Bookmark and URL Access” in Entitlement Key Reference . With the appropriate entitlements, you can create a security-scoped bookmark by calling the bookmarkDataWithOptions:includingResourceValuesForKeys:relativeToURL:error: method of the NSURL class (or its Core Foundation equivalent, the CFURLCreateBookmarkData function). When you later need access to a bookmarked resource, resolve its security-scoped bookmark by calling the the URLByResolvingBookmarkData:options:relativeToURL:bookmarkDataIsStale:error:method of the NSURL class (or its Core Foundation equivalent, the CFURLCreateByResolvingBookmarkData function). In a sandboxed app, you cannot access the file-system resource that a security-scoped URL points to until you call the startAccessingSecurityScopedResource method (or its Core Foundation equivalent, the CFURLStartAccessingSecurityScopedResource function) on the URL. When you no longer need access to a resource that you obtained using security scope (typically, after you close the resource) you must call the stopAccessingSecurityScopedResource method (or its Core Foundation equivalent, the CFURLStopAccessingSecurityScopedResource function) on the resource’s URL. Calls to start and stop access are nestable on a per-process basis. This means that if your app calls the start method on a URL twice, to fully relinquish access to the referenced resource you must call the corresponding stop method twice. If you call the stop method on a URL whose referenced resource you do not have access to, nothing happens. Warning: You must balance every call to the startAccessingSecurityScopedResource method with a corresponding call to the stopAccessingSecurityScopedResource method. If you fail to relinquish your access when you no longer need a file-system resource, your app leaks kernel resources. If sufficient kernel resources are leaked, your app loses its ability to add file-system locations to its sandbox, such as via Powerbox or security-scoped bookmarks, until relaunched. App Sandbox in Depth Security-Scoped Bookmarks and Persistent Resource Access 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 23For detailed descriptions of the methods, constants, and entitlementsto use for implementing security-scoped bookmarks in your app, read NSURL Class Reference or CFURL Reference , and read “Enabling Security-Scoped Bookmark and URL Access” in Entitlement Key Reference . App Sandbox and Code Signing After you enable App Sandbox and specify other entitlements for a target in your Xcode project, you must code sign the project. Take note of the distinction between how you set entitlements and how you set a code signing identity: ● Use the Xcode target editor to set entitlements on a target-by-target basis ● Use the Xcode project build settings to set the code signing identity for a project as a whole You must perform code signing because entitlements (including the special entitlement that enables App Sandbox) are built into an app’s code signature. From another perspective, an unsigned app is not sandboxed and has only default entitlements, regardless of settings you’ve applied in the Xcode target editor. OS X enforces a tie between an app’s container and the app’s code signature. This important security feature ensures that no other sandboxed app can access your container. The mechanism works as follows: After the system creates a container for an app, each time an app with the same bundle ID launches, the system checks that the app’s code signature matches a code signature expected by the container. If the system detects a mismatch, it prevents the app from launching. OS X’s enforcement of container integrity impacts your development and distribution cycle. This is because, in the course of creating and distributing an app, the app is code signed using various signatures. Here’s how the process works: 1. Before you create a project, you obtain two code signing certificatesfrom Apple: a development certificate and a distribution certificate. (To learn how to obtain code signing certificates, read “Creating Signing Certificates” in Tools Workflow Guide for Mac .) For development and testing, you sign your app with the development code signature. 2. When the Mac App Store distributes your app, it is signed with an Apple code signature. For testing and debugging, you may want to run both versions of your app: the version you sign and the version Apple signs. But OS X sees the Apple-signed version of your app as an intruder and won’t allow it to launch: Its code signature does not match the one expected by your app’s existing container. If you try to run the Apple-signed version of your app, you get a crash report containing a statement similar to this: App Sandbox in Depth App Sandbox and Code Signing 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 24Exception Type: EXC_BAD_INSTRUCTION (SIGILL) The solution is to adjust the access control list (ACL) on your app’s container to recognize the Apple-signed version of your app. Specifically, you add the designated code requirement of the Apple-signed version of your app to the app container’s ACL. To adjust an ACL to recognize an Apple-signed version of your app 1. Open Terminal (in /Applications/Utilities). 2. Open a Finder window that contains the Apple-signed version of your app. 3. In Terminal, enter the following command: asctl container acl add -file In place of the placeholder, substitute the path to the Apple-signed version of your app. Instead of manually typing the path, you can drag the app’s Finder icon to the Terminal window. The container’s ACL now includes the designated code requirements for both versions of your app. OS X then allows you to run either version of your app. You can use this same technique to share a container between (1) a version of an app that you initially signed with a self-generated code signature, such as the one you created in “App Sandbox Quick Start” (page 8), and (2) a later version that you signed with a development code signature from Apple. You can view the list of code requirements in a container’s ACL. For example, after adding the designated code requirement for the Apple-signed version of your app, you can confirm that the container’s ACL lists two permissible code requirements. To display the list of code requirements in a container’s ACL 1. Open Terminal (in /Applications/Utilities). 2. In Terminal, enter the following command: asctl container acl list -bundle In place of the placeholder,substitute the name of your app’s container directory. (The name of your app’s container directory is typically the same as your app’s bundle identifier.) App Sandbox in Depth App Sandbox and Code Signing 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 25For more information about working with App Sandbox container access control lists and their code requirements, read the man page for the asctl (App Sandbox control) tool. External Tools, XPC Services, and Privilege Separation Some app operations are more likely to be targets of malicious exploitation. Examples are the parsing of data received over a network, and the decoding of video frames. By using XPC, you can improve the effectiveness of the damage containment offered by App Sandbox by separating such potentially dangerous activities into their own address spaces. XPC is an OS X interprocess communication technology that complements App Sandbox by enabling privilege separation. Privilege separation, in turn, is a development strategy in which you divide an app into pieces according to the system resource access that each piece needs. The component pieces that you create are called XPC services. You create an XPC service as an individual target in your Xcode project. Each service gets its own sandbox—specifically, it gets its own container and its own set of entitlements. In addition, an XPC service that you include with your app is accessible only by your app. These advantages add up to making XPC the best technology for implementing privilege separation in an OS X app. By contrast, a child process created by using the posix_spawn function, by calling fork and exec (discouraged), or by using the NSTask class simply inherits the sandbox of the process that created it. You cannot configure a child process’s entitlements. For these reasons, child processes do not provide effective privilege separation. To use XPC with App Sandbox: ● Confer minimal privileges to each XPC service, according to its needs. ● Design the data transfers between the main app and each XPC service to be secure. ● Structure your app’s bundle appropriately. The life cycle of an XPC service, and its integration with Grand Central Dispatch (GCD), is managed entirely by the system. To obtain this support, you need only to structure your app’s bundle correctly. For more on XPC, see “Creating XPC Services” in Daemons and Services Programming Guide . App Sandbox in Depth External Tools, XPC Services, and Privilege Separation 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 26There’s a common, basic workflow for designing or converting an app for App Sandbox. The specific steps to take for your particular app, however, are as unique as your app. To create a work plan for adopting App Sandbox, use the process outlined here, along with the conceptual understanding you have from the earlier chapters in this document. Six Steps for Adopting App Sandbox The workflow to convert an OS X app to work in a sandbox typically consists of the following six steps: 1. Determine whether your app is suitable for sandboxing. 2. Design a development and distribution strategy. 3. Resolve API incompatibilities. 4. Apply the App Sandbox entitlements you need. 5. Add privilege separation using XPC. 6. Implement a migration strategy. Determine Whether Your App Is Suitable for Sandboxing Most OS X apps are fully compatible with App Sandbox. If you need behavior in your app that App Sandbox does not allow, consider an alternative approach. For example, if your app depends on hard-coded paths to locationsin the user’s home directory, consider the advantages of using Cocoa and Core Foundation path-finding APIs, which use the sandbox container instead. If you choose to not sandbox your app now, or if you determine that you need a temporary exception entitlement, use Apple’s bug reporting system to let Apple know what’s not working for you. Apple considers feature requests as it develops the OS X platform. Also, if you request a temporary exception, be sure to use the Review Notes field in iTunes Connect to explain why the exception is needed. The following app behaviors are incompatible with App Sandbox: ● Use of Authorization Services 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 27 Designing for App SandboxWith App Sandbox, you cannot do work with the functions described in Authorization Services C Reference . ● Use of accessibility APIs in assistive apps With App Sandbox, you can and should enable your app for accessibility, as described in Accessibility Overview for OS X . However, you cannot sandbox an assistive app such as a screen reader, and you cannot sandbox an app that controls another app. ● Sending Apple events to arbitrary apps With App Sandbox, you can receive Apple events and respond to Apple events, but you cannotsend Apple events to arbitrary apps. By using a temporary exception entitlement, you can enable the sending of Apple eventsto a list ofspecific apps that you specify, as described in Entitlement Key Reference . ● Sending user-info dictionaries in broadcast notifications to other tasks With App Sandbox, you cannot include a user-info dictionary when posting to an NSDistributedNotificationCenter object for messaging other tasks. (You can , as usual, include a user-info dictionary when messaging other parts of your app by way of posting to an NSNotificationCenter object.) ● Loading kernel extensions Loading of kernel extensions is prohibited with App Sandbox. ● Simulation of user input in Open and Save dialogs If your app depends on programmatically manipulating Open or Save dialogs to simulate or alter user input, your app is unsuitable for sandboxing. ● Setting preferences on other apps With App Sandbox, each app maintains its preferences inside its container. Your app has no access to the preferences of other apps. ● Configuring network settings With App Sandbox, your app cannot modify the system’s network configuration (whether with the System Configuration framework, the CoreWLAN framework, or other similar APIs) because doing so requires administrator privileges. ● Terminating other apps With App Sandbox, you cannot use the NSRunningApplication class to terminate other apps. Designing for App Sandbox Determine Whether Your App Is Suitable for Sandboxing 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 28Design a Development and Distribution Strategy During development, you may have occasion to run versions of your app that are signed with different code signatures. After you’ve run your app signed using one signature, the system won’t allow a second version of your app, signed with a second signature, to launch—unless you modify the app’s container. Be sure to understand how to handle this, as described in “App Sandbox and Code Signing” (page 24), as you design your development strategy. When a customer first launches a sandboxed version of your app, the system creates a container for your app. The access control list (ACL) for the container is established at that time, and the ACL istied to the code signature of that version of your app. The implication for you is that all future versions of the app that you distribute must use the same code signature. To learn how to obtain code signing certificatesfrom Apple, read “Creating Signing Certificates” in Tools Workflow Guide for Mac . Resolve API Incompatibilities If you are using OS X APIs in ways that were not intended, or in ways that expose user data to attack, you may encounter incompatibilities with App Sandbox. This section provides some examples of app design that are incompatible with App Sandbox and suggests what you can do instead. Opening, Saving, and Tracking Documents If you are managing documents using any technology other than the NSDocument class, you should convert to using this classto benefit from its built-in App Sandbox support. The NSDocument class automatically works with Powerbox. NSDocument also provides support for keeping documents within your sandbox if the user moves them using the Finder. Remember that the inheritance path of the NSOpenPanel and NSSavePanel classes is different when your app is sandboxed. See “Open and Save Dialog Behavior with App Sandbox” (page 19). If you don’t use the NSDocument class to manage your app’s documents, you can craft your own file-system support for App Sandbox by using the NSFileCoordinator class and the NSFilePresenter protocol. Retaining Access to File System Resources If your app depends on persistent access to file system resources outside of your app’s container, you need to adopt security-scoped bookmarks as described in “Security-Scoped Bookmarks and Persistent Resource Access” (page 21). Designing for App Sandbox Design a Development and Distribution Strategy 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 29Creating a Login Item for Your App To create a login item for your sandboxed app, use the SMLoginItemSetEnabled function (declared in ServiceManagement/SMLoginItem.h) as described in “Adding Login Items Using the Service Management Framework” in Daemons and Services Programming Guide . (With App Sandbox, you cannot create a login item using functions in the LSSharedFileList.h header file. For example, you cannot use the function LSSharedFileListInsertItemURL. Nor can you manipulate the state of launch services, such as by using the function LSRegisterURL.) Accessing User Data OS X path-finding APIs, above the POSIX layer, return paths relative to the container instead of relative to the user’s home directory. If your app, before you sandbox it, accesses locations in the user’s actual home directory (~) and you are using Cocoa or Core Foundation APIs, then, after you enable sandboxing, your path-finding code automatically uses your app’s container instead. For first launch of your sandboxed app, OS X automatically migrates your app’s main preferences file. If your app uses additional support files, perform a one-time migration of those files to the container, as described in “Migrating an App to a Sandbox” (page 33). If you are using a POSIX function such as getpwuid to obtain the path to the user’s actual home directory, consider instead using a Cocoa or Core Foundation symbol such as the NSHomeDirectory function. By using Cocoa or Core Foundation, you support the App Sandbox restriction against directly accessing the user’s home directory. If your app requires access to the user’s home directory in order to function, let Apple know about your needs using the Apple bug reporting system. In addition, be sure to follow the guidance regarding entitlements provided on the iTunes Connect website. Accessing Preferences of Other Apps Because App Sandbox directs path-finding APIs to the container for your app, reading or writing to the user’s preferencestakes place within the container. Preferencesfor othersandboxed apps are inaccessible. Preferences for appsthat are notsandboxed are placed in the ~/Library/Preferences directory, which is also inaccessible to your sandboxed app. If your app requires access to another app’s preferences in order to function—for example, if it requires access to the playlists that a user has defined for iTunes—let Apple know about your needs using the Apple bug reporting system. In addition, be sure to follow the guidance regarding entitlements provided on the iTunes Connect website. Designing for App Sandbox Resolve API Incompatibilities 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 30With these provisosin mind, you can use a path-based temporary exception entitlement to gain programmatic accessto the user’s ~/Library/Preferences folder. Use a read-only entitlement to avoid opening the user’s preferences to malicious exploitation. A POSIX function, such as getpwuid, can provide the file system path you need. For details on entitlements, see Entitlement Key Reference . Apply the App Sandbox Entitlements You Need To adopt App Sandbox for a target in an Xcode project, apply the value to the com.apple.security.app-sandbox entitlement key for that target. Do this in the Xcode target editor by selecting the Enable App Sandboxing checkbox. Apply other entitlements as needed. For a complete list, refer to Entitlement Key Reference . Important: App Sandbox protects user data most effectively when you minimize the entitlements you request. Take care not to request entitlements for privileges your app does not need. Consider whether making a change in your app could eliminate the need for an entitlement. Here’s a basic workflow to use to determine which entitlements you need: 1. Run your app and exercise its features. 2. In the Console app (available in /Applications/Utilities/), look for sandboxd violations in the All Messages system log query. Each such violation indicates that your app attempted to do something not allowed by your sandbox. Here’s what a sandboxd violation looks like in Console: 3:56:16 pm sandboxd: ([4928]) AppSandboxQuickS(4928) deny network-outbound 111.30.222.15:80 3:56:16 pm sandboxd: ([4928]) AppSandboxQuickS(4928) deny system-socket Click the paperclip icon to the right of a violation message to view the backtrace that shows what led to the violation. 3. For each sandboxd violation you find, determine how to resolve the problem. In same cases, a simple change to your app,such as using your Container instead of other file system locations,solvesthe problem. In other cases, applying an App Sandbox entitlement using the Xcode target editor is the best choice. 4. Using the Xcode target editor, enable the entitlement that you think will resolve the violation. 5. Run the app and exercise its features again. Either confirm that you have resolved the sandboxd violation, or investigate further. Designing for App Sandbox Apply the App Sandbox Entitlements You Need 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 31If you choose not to sandbox your app now or to use a temporary exception entitlement, use Apple’s bug reporting system to let Apple know about the issue you are encountering. Apple considers feature requests as it develops the OS X platform. Also, be sure use the Review Notes field in iTunes Connect to explain why the exception is needed. Add Privilege Separation Using XPC When developing for App Sandbox, look at your app’s behaviors in terms of privileges and access. Consider the potential benefitsto security and robustness ofseparating high-risk operationsinto their own XPC services. When you determine that a feature should be placed into an XPC service, do so by referring to “Creating XPC Services” in Daemons and Services Programming Guide . Implement a Migration Strategy Ensure that customers who are currently using a pre-sandbox version of your app experience a painless upgrade when they install the sandboxed version. For details on how to implement a container migration manifest, read “Migrating an App to a Sandbox” (page 33). Designing for App Sandbox Add Privilege Separation Using XPC 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 32An app that is not sandboxed places its support files in locations that are inaccessible to a sandboxed version of the same app. For example, the typical locations for support files are shown here: Path Description Legacy location ~/Library/Application Support// Sandbox location ~/Library/Containers//Data/Library/Application Support// As you can see, the sandbox location for the Application Support directory is within an app’s container—thus allowing the sandboxed app unrestricted read/write access to those files. If you previously distributed your app without sandboxing and you now want to provide a sandboxed version, you must move support files into their new, sandbox-accessible locations. Note: The system automatically migrates your app’s preferences file (~/Library/Preferences/com.yourCompany.YourApp.plist) on firstlaunch of yoursandboxed app. OS X provides support-file migration, on a per-user basis, when a user first launches the sandboxed version of your app. This support depends on a special property list file you create, called a container migration manifest. A container migration manifest consists of an array of strings that identify the support files and directories you want to migrate when a user first launches the sandboxed version of your app. The file’s name must be container-migration.plist. For each file or directory you specify for migration, you have a choice of allowing the system to place the item appropriately in your container, or explicitly specifying the destination location. OS X moves—it does not copy—the files and directories you specify in a container migration manifest. That is, the files and directories migrated into your app’s container no longer exist at their original locations. In addition, container migration is a one-way process: You are responsible for providing a way to undo it, should you need to do so during development or testing. The section “Undoing a Migration for Testing” (page 36) provides a suggestion about this. 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 33 Migrating an App to a SandboxCreating a Container Migration Manifest To support migration of app support files when a user first launches the sandboxed version of your app, create a container migration manifest. To create and add a container migration manifest to an Xcode project 1. Add a property list file to the Xcode project. The Property List template is in the OS X “Resource” group in the file template dialog. Important: Be sure to name the file container-migration.plist spelled and lowercased exactly this way. 2. Add a Move property to the container migration manifest. The Move property is the lone top-level key in a container migration manifest. You add it to the empty file as follows: ● Right-click the empty editor for the new .plist file, then choose Add Row. ● In the Key column, enter Move as the name of the key. You must use this exact casing and spelling. ● In the Type column, choose Array. 3. Add a string to the Move array for the first file or folder you want to migrate. For example, suppose you want to migrate your Application Support directory (along with its contained files and subdirectories) to your container. If your directory is called App Sandbox Quick Start and is currently within the ~/Library/Application Support directory, use the following string as the value for the new property list item: ${ApplicationSupport}/App Sandbox Quick Start No trailing slash character is required, and space characters are permitted. The search-path constant in the path is equivalent to ~/Library/Application Support. This constant is described, along with other commonly used directories, in “Use Variables to Specify Support-File Directories” (page 37). Similarly, add additional strings to identify the original (before sandboxing) paths of additional files or folders you want to migrate. Migrating an App to a Sandbox Creating a Container Migration Manifest 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 34When you specify a directory to be moved, keep in mind that the move is recursive—it includes all the subdirectories and files within the directory you specify. Before you first test a migration manifest, provide a way to undo the migration, such as suggested in “Undoing a Migration for Testing” (page 36). To test a container migration manifest 1. In the Finder, open two windows as follows: ● In one window, view the contents of the ~/Library/Containers/ directory. ● In the other window, view the contents of the directory containing the support files named in the container migration manifest—that is, the files you want to migrate. 2. Build and run the Xcode project. Upon successful migration, the support files disappear from the original (nonsandbox) directory and appear in your app’s container. If you want to alter the arrangement ofsupport files during migration, use a slightly more complicated .plist structure. Specifically, for a file or directory whose migration destination you want to control, provide both a starting and an ending path. The ending path is relative to the Data directory in your container. In specifying an ending path, you can use any of the search-path constants described in “Use Variablesto Specify Support-File Directories” (page 37). If your destination path specifies a custom directory (one that isn’t part of a standard container), the system creates the directory during migration. The following task assumes that you’re using the Xcode property list editor and working with the container migration manifest you created earlier in this chapter. To control the destination of a migrated file or directory 1. In the container migration manifest, add a new item to the Move array. 2. In the Type column, choose Array. 3. Add two strings as children of the new array item. 4. In the top string of the pair, specify the origin path of the file or directory you want to migrate. 5. In the bottom string of the pair, specify the destination (sandbox) custom path for the file or directory you want to migrate. Migrating an App to a Sandbox Creating a Container Migration Manifest 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 35File migration proceeds from top-to-bottom through the container migration manifest. Take care to list items in an order that works. For example,suppose you want to move your entire Application Support directory as-is, except for one file. You want that file to go into a new directory parallel to Application Support in the container. For this approach to work, you must specify the individual file move before you specify the move of the Application Support directory—that is, specify the individual file move higher in the container migration manifest. (If Application Support were specified to be moved first, the individual file would no longer be at its original location at the time the migration process attempted to move it to its new, custom location in the container.) Undoing a Migration for Testing When testing migration of support files, you may find it necessary to perform migration more than once. To support this, you need a way to restore your starting directory structures—that is, the structures as they exist prior to migration. One way to do this is to make a copy of the directories to migrate, before you perform a first migration. Save this copy in a location unaffected by the migration manifest. The following task assumes you have created this sort of backup copy. To manually undo a container migration for testing purposes 1. Manually copy the files and directories—those specified in the manifest—from your backup copy to their original (premigration) locations. 2. Delete your app’s container. The next time you launch the app, the system recreates the container and migrates the support files according to the current version of the container migration manifest. An Example Container Migration Manifest Listing 4-1 shows an example manifest as viewed in a text editor. Listing 4-1 An example container migration manifest Migrating an App to a Sandbox Undoing a Migration for Testing 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 36 Move ${Library}/MyApp/MyConfiguration.plist ${Library}/MyApp/MyDataStore.xml ${ApplicationSupport}/MyApp/MyDataStore.xml This manifest specifies the migration of two items from the user’s Library directory to the app’s container. For the first item, MyConfiguration.plist, only the origin path is specified, leaving it to the migration process to place the file appropriately. For the second item, MyDataStore.xml, both an origin and a custom destination path are specified. The ${Library} and ${ApplicationSupport} portions of the paths are variables you can use as a convenience. For a list of variables you can use in a container migration manifest, see “Use Variables to Specify Support-File Directories” (page 37). Use Variables to Specify Support-File Directories When you specify a path in a container migration manifest, you can use certain variables that correspond to commonly used support file directories. These variables work in origin and destination paths, but the path that a variable resolves to depends on the context. Refer to Table 4-1. Table 4-1 How system directory variables resolve depending on context Context Variable resolves to Origin path Home-relative path (relative to the ~ directory) Destination path Container-relative path (relative to the Data directory in the container) Migrating an App to a Sandbox Use Variables to Specify Support-File Directories 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 37The variables you can use for specifying support-file directories are described in Table 4-2 (page 38). For an example of how to use these variables, see Listing 4-1 (page 36). You can also use a special variable that resolves to your app’s bundle identifier, allowing you to conveniently incorporate it into an origin or destination path. This variable is ${BundleId}. Table 4-2 Variables for support-file directories Variable Directory The directory containing application support files. Corresponds to the NSApplicationSupportDirectory search-path constant. ${ApplicationSupport} The directory containing the user’s autosaved documents. Corresponds to the NSAutosavedInformationDirectory search-path constant. ${AutosavedInformation} The directory containing discardable cache files. Corresponds to the NSCachesDirectory search-path constant. ${Caches} Each variable correspondsto the directory containing the user’s documents. Corresponds to the NSDocumentDirectory search-path constant. ${Document} ${Documents} The current user’s home directory. Corresponds to the directory returned by the NSHomeDirectory function. When in a destination path in a manifest, resolves to the Container directory. ${Home} The directory containing application-related support and configuration files. Corresponds to the NSLibraryDirectory search-path constant. ${Library} Migrating an App to a Sandbox Use Variables to Specify Support-File Directories 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 38This table describes the changes to App Sandbox Design Guide . Date Notes 2012-09-19 Clarified information about launching external tools. 2012-07-23 Added an explanation of app group containers. Improved the explanation of security-scoped bookmarks in “Security-Scoped Bookmarks and Persistent Resource Access” (page 21); updated that section for OS X v10.7.4. 2012-05-14 Added a brief section in the “Designing for App Sandbox” chapter: “Retaining Access to File System Resources” (page 29). Improved the discussion in “Opening, Saving, and Tracking Documents” (page 29), adding information about using file coordinators. Corrected the information in “Creating a Login Item for Your App” (page 30). Improved explanation ofsecurity-scoped bookmarksin “Security-Scoped Bookmarks and Persistent Resource Access” (page 21). 2012-03-14 Clarified the explanation of the container directory in “The App Sandbox Container Directory” (page 16) Updated for OS X v10.7.3, including an explanation of how to use security-scoped bookmarks. 2012-02-16 Added a section explaining how to provide persistent accessto file-system resources, “Security-Scoped Bookmarks and Persistent Resource Access” (page 21). 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 39 Document Revision HistoryDate Notes Expanded the discussion in “Powerbox and File System Access Outside of Your Container” (page 17) to better explain how user actions expand your app’s file system access. Added a section detailing the changes in behavior of Open and Save dialogs, “Open and Save Dialog Behavior with App Sandbox” (page 19). New document that explains Apple's security technology for damage containment, and how to use it. 2011-09-27 Portions of this document were previously published in Code Signing and Application Sandboxing Guide . Document Revision History 2012-09-19 | © 2012 Apple Inc. All Rights Reserved. 40Apple Inc. © 2012 Apple Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrievalsystem, or transmitted, in any form or by any means, mechanical, electronic, photocopying, recording, or otherwise, without prior written permission of Apple Inc., with the following exceptions: Any person is hereby authorized to store documentation on a single computer for personal use only and to print copies of documentation for personal use provided that the documentation contains Apple’s copyright notice. No licenses, express or implied, are granted with respect to any of the technology described in this document. Apple retains all intellectual property rights associated with the technology described in this document. This document is intended to assist application developers to develop applications only for Apple-labeled computers. Apple Inc. 1 Infinite Loop Cupertino, CA 95014 408-996-1010 Apple, the Apple logo, Cocoa, Finder, iTunes, Keychain, Mac, OS X, Sand, and Xcode are trademarks of Apple Inc., registered in the U.S. and other countries. QuickStart is a trademark of Apple Inc. iCloud is a service mark of Apple Inc., registered in the U.S. and other countries. App Store and Mac App Store are service marks of Apple Inc. iOS is a trademark or registered trademark of Cisco in the U.S. and other countries and is used under license. Even though Apple has reviewed this document, APPLE MAKES NO WARRANTY OR REPRESENTATION, EITHER EXPRESS OR IMPLIED, WITH RESPECT TO THIS DOCUMENT, ITS QUALITY, ACCURACY, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.ASARESULT, THISDOCUMENT IS PROVIDED “AS IS,” AND YOU, THE READER, ARE ASSUMING THE ENTIRE RISK AS TO ITS QUALITY AND ACCURACY. IN NO EVENT WILL APPLE BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL,OR CONSEQUENTIAL DAMAGES RESULTING FROM ANY DEFECT OR INACCURACY IN THIS DOCUMENT, even if advised of the possibility of such damages. THE WARRANTY AND REMEDIES SET FORTH ABOVE ARE EXCLUSIVE AND IN LIEU OF ALL OTHERS, ORAL OR WRITTEN, EXPRESS OR IMPLIED. No Apple dealer, agent, or employee is authorized to make any modification, extension, or addition to this warranty. Some states do not allow the exclusion or limitation of implied warranties or liability for incidental or consequential damages, so the above limitation or exclusion may not apply to you. This warranty gives you specific legal rights, and you may also have other rights which vary from state to state. iTunes Connect Sales and Trends Guide App Store  Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 1Apple Inc. © 2012 Apple Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, mechanical, electronic, photocopying, recording, or otherwise, without prior written permission of Apple Inc., with the following exceptions: Any person is hereby authorized to store documentation on a single computer for personal use only and to print copies of documentation for personal use provided that the documentation contains Apple’s copyright notice. The Apple logo is a trademark of Apple Inc. Use of the “keyboard” Apple logo (Option-Shift-K) for commercial purposes without the prior written consent of Apple may constitute trademark infringement and unfair competition in violation of federal and state laws. No licenses, express or implied, are granted with respect to any of the technology described in this document. Apple retains all intellectual property rights associated with the technology described in this document. This document is intended to assist partners in understanding the Sales and Trends module of iTunes Connect. Every effort has been made to ensure that the information in this document is accurate. Apple is not responsible for typographical errors. Apple Inc. 1 Infinite Loop Cupertino, CA 95014 408-996-1010 Even though Apple has reviewed this document, APPLE MAKES NO WARRANTY OR REPRESENTATION, EITHER EXPRESS OR IMPLIED, WITH RESPECT TO THIS DOCUMENT, ITS QUALITY, ACCURACY, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. AS A RESULT, THIS DOCUMENT IS PROVIDED “AS IS,” AND YOU, THE READER, ARE ASSUMING THE ENTIRE RISK AS TO ITS QUALITY AND ACCURACY. IN NO EVENT WILL APPLE BE LIABLE FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES RESULTING FROM ANY DEFECT OR INACCURACY IN THIS DOCUMENT, even if advised of the possibility of such damages THE WARRANTY AND REMEDIES SET FORTH ABOVE ARE EXCLUSIVE AND IN LIEU OF ALL OTHERS, ORAL OR WRITTEN, EXPRESS OR IMPLIED. No Apple dealer, agent, or employee is authorized to make any modification, extension, or addition to this warranty. Some states do not allow the exclusion or limitation of implied warranties or liability for incidental or consequential damages, so the above limitation or exclusion may not apply to you. This warranty gives you specific legal rights, and you may also have other rights which vary from state to state. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 2Contents 1. Getting Started 4 2. Navigating and Viewing Your Sales and Trends Data 5 2.1. Dashboard View 6 2.2. Sales View 11 3. Downloading, Reading and Understanding Sales and Trends Data 13 3.1. Downloading Reports 13 3.2. Auto-Ingest Tool 14 3.3. Reading Reports 16 3.4. Understanding Units 18 4. Contact Us 19 Appendix A - Sales Report Field Definitions 20 Appendix B - Opt-In Report Field Definitions 21 Appendix C - Apple Fiscal Calendar 22 Appendix D - Definition of Day and Week 23 Appendix E – Product Type Identifiers 24 Appendix F – Country Codes 25 Appendix G – Promotional Codes 26 Appendix H – Currency Codes 27 Appendix I - Subscription and Period Field Values 28 Appendix J - FAQs 29 Appendix K - Sample Sales Report 30 Appendix L – Other Uses 32 Appendix M - Newsstand Report Field Definitions 33 Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 31. Getting Started iTunes Connect can be accessed at http://itunesconnect.apple.com. Once you login, you will be presented with the Welcome page below, which contains notifications at the top and module links to help you navigate through iTunes Connect. The Welcome page you will see is based on the modules applicable to you and may be different from what is shown below. This guide is primarily intended to cover the Sales and Trends module. The initial user who entered into the program license agreement has the “Admin” role, which provides access to all modules, including the ability to add other “Admin” users (using the Manage Users module). The “Admin” users associated with your account are expected to manage (add, modify, and delete) your users based on your needs. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 42. Navigating and Viewing Your Sales and Trends Data The iTunes Connect Sales and Trends module allows you to interact with your sales data in various ways: ■ A summary that provides total units, percent differences, graphs, top selling content and largest market information (Dashboard view). ■ Previews that provide the top 50 transactions of sales aggregated at the title level in descending sorted order (Sales view). ■ Download full transaction reports for import and further analysis (Sales view). When you are ready to access the Sales and Trends module, click on the following link located on the Welcome page: Upon selecting the Sales and Trends module, you will be taken to the Dashboard view. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 52.1. Dashboard View The Dashboard will load and display the most recent daily data available. The following identifies the various components of the dashboard. The “Selection” controls located above the graph allow you to change the information displayed. Vendor Selection The Vendor Selection display lists the legal entity name for the Sales and Trends that you are viewing. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 6View Selection The View Selection allows you to switch between different views. In addition to the Dashboard view, you can toggle to the Sales view (the Sales view is covered in section 2.2). Period Selection You can choose the type (daily or weekly), as well as the period of interest. The date menu will display all periods available up to the last 13 weeks or 14 days. Category Selection You can choose the specific category you wish to view in the Dashboard if you sell more than one type of content (i.e. iOS and MacOS). Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 7Type Selection You can choose the specific type of content within a category to view in the Dashboard’s graph, Top Products and Top Markets. The available types are the same for both the iOS and MacOS category. Refer to Appendix E for the complete product breakdown by product type. Graph Selection You can choose between a line graph and bar graph by clicking on the graph buttons located on the right top corner of the graph. Graph The data displayed in the graph is based on the period (specific day or week), category and type selected. When you hover over a specific day or week in the graph (bar or line), the date, number of units and type will be displayed. The following displays the graph for the period of August 30, 2010 and the Free Apps category while mousing over the August 30, 2010 bar. When viewing daily reports, the graph will also display the percentage change from the same day in the prior period. In the graph above you see the percentage change of free apps sold on 8/30 (Monday) to those sold on 8/24 (Monday of prior week) based on units. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 8Top Products Display The Top Products display is based on the period (specific day or week), category (iOS or Mac OS) and the type (Free Apps, Paid Apps, In Apps, Updates) selected. The section provides a summary of net units at the Product level. A Product can be reported as separate lines in your reports due to differences such as territories but will be reported as combined in terms of units in this display since the units are aggregated at the Product level world-wide based on unique product identifier. The “Change” column in the display shows units and percentage change from the prior period (selected day over same day of the prior week, or selected week over prior week). Top Markets Display The Top Markets display is based on the period (specific day or week), category (iOS and Mac OS) and the type (e.g. Free Apps) selected. This section provides a summary of net units for all products at the country (iTunes Storefront) level. The “Change” column in the display shows units and percentage change from the prior period (selected day over same day of the prior week, or selected week over prior week). See Appendix F for iTunes Storefront listing. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 9Resources At the bottom left of all pages you will find three links: ■ Fiscal Calendar - Opens a new window that displays Apple’s fiscal calendar ■ User Guide - Provides the most current version of this guide ■ Mobile Guide - Provides the user guide for the iTC Mobile Application. Done Button The “Done” button at the bottom right of all pages takes you to the Dashboard from the Sales view, and to the iTunes Connect Welcome page from the Dashboard. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 102.2. Sales View The Sales view allows you to analyze at the specific content level. You can preview the Top 50 products delivered based on transaction volume summarized and sorted descending by Units, and can download the available daily and weekly reports for additional information about all your transactions. You can also download detailed Newsstand reports or contact information for customers that have elected to “opt-in” when purchasing an In-App Purchase subscription. The following is an example of the Sales view. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 11Understanding The Sales Preview When you land on the Sales view, the Period presented is the latest daily data available. Using the Period Selection menu, you can preview all available daily and weekly data for all content types in all categories. Once you have selected a period, the Preview will be displayed. The Preview summarizes the data based on the columns displayed, including any promotional transactions indicated with (Promo Indicator). You can hover over the Promo Indicator to see the type of promotion. See Appendix G for Promotional Codes. Autorenewable subscription transactions are indicated with (Subscription Indicator). The preview functionality does not contain the full report. To view or analyze all transactions you must download the full reports. The previews summarize data differently than the reports based on the information available (i.e. the preview may summarize sales at a higher level as the downloaded report has more fields to consider). Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 123. Downloading, Reading and Understanding Sales and Trends Data 3.1. Downloading Reports You may download the Sales reports from the respective Sales view. To download a report (tab delimited zipped text file), you must select a report period (day of week or week ended) and press the download button to the right of the period selection menu. For a complete listing of fields please see Appendix A, B and M. If you are using Mac OS X the reports will automatically open when downloaded. If you are using a Windows OS you will need to download an application (for example WinZip) to decompress the “.gz” file prior to use. You can then import the tab delimited text file to a database or spreadsheet application (Numbers, MS Excel) and analyze or manipulate your data as needed. Weekly reports cover Monday through Sunday and are available on Mondays. The daily reports represent the 24 hour period in the time zone of the respective storefront (territory). Please refer to Appendix D for the definition of Day and Week. We do not store or regenerate the data after the periods have expired (14 rolling days and 13 rolling weeks); you will need to download and store this data on a regular basis if you intend to use it in the future. Downloading Customer Opt-In Information If your apps have auto-renewable subscriptions, you can download contact information for customers who have elected to “opt-in” to personal information sharing. To download the report (tab delimited zipped text file), you must select a weekly report period and click Opt-In Report next to Download Report. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 13To open the encrypted .zip file that is downloaded, you need to use the Opt-In Passkey. To obtain the Opt-In Passkey, click the Opt-In Passkey button in the upper right of the screen. The passkey will be displayed in a lightbox. Copy and paste this value to use it to unpack the .zip file and access the Opt-In Report. You will need to use a decompression tool like Stuff-It Expander or Winzip to open the encrypted file once you have downloaded it. Downloading Newsstand Reports If you have one or more Newsstand apps available for sale, you can download Newsstand reports by clicking Newsstand Detailed. Newsstand reports are also available via auto-ingest. 3.2. Auto-Ingest Tool Apple provides access to a Java based tool to allow you to automate the download of your iTunes Connect Sales and Trends reports. To use the auto-ingest tool, configuration on your part will be required. This tool allows you to automate the retrieval of: •Daily Summary Reports •Weekly Summary Reports •Opt-In Reports •Newsstand Reports As new reports become available we will modify and redeliver the java package or new parameters to use to download new products (i.e. we will modify the script for new features). We will communicate both the anticipated date of the report release and the date that the tool will be able to retrieve reports. You may not alter or disseminate the auto-ingest tool for any reason. We reserve the right to revoke access for usage or distribution beyond its intended use. Auto-Ingest Instructions You must have Java installed on the machine where you are running the auto-ingest tool. The tool will work as expected with Java version 1.6 or above. Follow the steps below to setup the environment for auto-ingestion: Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 141. Download and save the file Autoingestion.class to the directory where you want the reports delivered. http://www.apple.com/itunesnews/docs/Autoingestion.class.zip 2. To run the Java class file, change the command line directory to the directory where the class file is stored. 3. Invoke the following from the command line: java Autoingestion All items contained within “< >” are variable and will require you to define them. Of the 7 parameters only the date is optional. If you do not put a date in the parameter we will provide you the latest available report (the other parameters are mandatory). You will need to delimit the parameters with a space. Parameters Definitions Variable Value Notes username Your user name The user name you use to log into iTunes Connect password Your password The password you use to log into iTunes Connect vendorid 8####### (your unique number) Vendor ID for the entity which you want to download the report report_type Sales or Newsstand This is the report type you want to download. date_type Daily or Weekly Selecting Weekly will provide you the Weekly version of the report. Selecting Daily will provide you the Daily version of the report. report_subtype Summary, Detailed or Opt-In This is the parameter for the Sales Reports. Note: Detailed can only be used for Newsstand reports. Date (optional) YYYYMMDD This is the date of report you are requesting. If the value for Date parameter is not provided, you will get the latest report available. Example: You access iTunes Connect with user name “john@xyz.com” and your password is “letmein” for company 80012345, and you want to download a sales - daily - summary report for February 4, 2010. You will need to invoke the job by running the following command from the directory where the class file is stored: java Autoingestion john@xyz.com letmein 80012345 Sales Daily Summary 20100204 Newsstand Reports via Auto-Ingest If you are using auto-ingest, you can create the reports using the following auto-ingest parameters: Daily java Autoingestion Newsstand Daily Detailed java Autoingestion N D D java Autoingestion 5 2 1 Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 15Weekly java Autoingestion Newsstand Weekly Detailed java Autoingestion N W D java Autoingestion 5 1 1 3.3. Reading Reports Report File Names The file names for downloaded reports follow a standard naming convention. Please refer to the matrix below for details. Report Data Report Type Reporting Range Naming Convention Description Sales Summary Daily S_D__ Example: S_D_80000000_20111104 The first letter identifies that the report provides Sales data at a Summary level. Second letter denotes that it is a Daily report. This is followed by the Vendor Account Number and the Date of reporting data ('YYYYMMDD'). Sales Summary Weekly S_W__ Example: S_W_80000000_20111104 The first letter identifies that the report provides Sales data at a Summary level. Second letter denotes that it is a Weekly report. This is followed by the Vendor Account Number and the Date of reporting data ('YYYYMMDD'). Opt_in Summary Weekly O_S_W__ Example: O_S_W_80000000_20111104 The first and second letters identify that the report provides customer Opt-in data at a Summary level. The third letter identifies that it is a Weekly report. This is followed by the Vendor Account Number and the Date of reporting data ('YYYYMMDD'). Newsstand Detailed Daily N_D_D__ Example: N_D_D_80000000_20111104 The first and second letters identify that the report provides customer Newsstand data at a Detailed level. The third letter identifies that it is a Daily report. This is followed by the Vendor Account Number and the Date of reporting data ('YYYYMMDD'). Newsstand Detailed Weekly N_D_W__ Example: N_D_W_80000000_20111104 The first and second letters identify that the report provides customer Newsstand data at a Detailed level. The third letter identifies that it is a Weekly report. This is followed by the Vendor Account Number and the Date of reporting data ('YYYYMMDD'). Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 16Report Field Names All reports have a header row which contains the field names of each column. The reports present transactions that can be tracked with your SKU and/or the Apple Identifier. For a complete listing of fields please see Appendix A, B and M. Key Field Mapping The following table and screenshots will help you understand which fields in the report were setup by you in iTunes Connect and where they are in the App Store: Reference Field Name In Report Field in iTunes Connect Field in App Store 1 Developer Company Name Displayed after genre 2 Title App Name Displayed at top of product page 3 SKU SKU Number Not displayed on App Store Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 17Apple Identifier The Apple Identifier is the unique product identifier assigned by Apple. It is always included in each row of your sales reports. We recommend you provide the Apple Identifier of your app whenever you contact us for support so that your request can be expedited. You can also access the Apple Identifier by using the links in the App Store: The menu will offer an option for “Copy Link”. The link will look like the link below. The string of numbers highlighted is the Apple Identifier of the app. http://itunes.apple.com/us/app/remote/id284417350?mt=85 3.4. Understanding Units The reports are designed to provide valuable information about the activity of your product on the App Store. This can result in many lines for a given product. For each product with a unique Apple Identifier and SKU, units are split by: ■ Storefront / Country Code (US, UK) ■ Sales vs. Refunds ■ Product Type ■ Price ■ Promo Code ■ App Version Here are some examples of how units are grouped and displayed in both Preview and downloaded reports. Example 1: If you have one product and you are selling the product in the US, you will see 1 row (1 for US sales) assuming there are no refunds, price changes and promo codes during the period. Example 2: If you are selling 30 products in the US, and 10 of the products have refunds, then the preview and the downloaded report will have 40 rows and you will see a row for sales and a row for refunds. Example 3: If you are selling 30 products in the US, and 5 products have a price change in the middle of the week, then your full report and your previews will have 35 rows and you will see 2 lines per app with a price change. Example 4: If 10 new customers purchase your app and 10 existing customers update to the latest version of your app in the US, then your preview and downloaded report will have 1 row for purchases and 1 row for updates. Example 5: If 10 customers purchase version 1.1 of your product in the US, and those customers then update to version 1.2 of the same product, then your preview and downloaded report will have 2 rows, 1 row for purchases of version 1.1 and 1 row for updates to version 1.2. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 184. Contact Us If you have any questions or have difficulties viewing or downloading your sales and trends information, please do not hesitate to contact us. The easiest way to ensure your request is routed correctly is to use the Contact Us module. A Contact Us link is available on all pages as part of the footer. You can also find the Contact Us module on the iTunes Connect Homepage: The link will take you to a page that allows you to select the topic you need help with and will ask a series of questions and provide answers where available. For Sales and Trends inquiries, select the “Sales and Trends” topic. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 19Appendix A - Sales Report Field Definitions The definitions apply to Daily and Weekly Reports. Report Field Report Data Type Values Notes Provider CHAR(5) - APPLE Up to 5 Characters The service provider in your reports will typically be Apple Provider Country CHAR(2) - US Up to 2 Characters The service provider country code will typically be US SKU VARCHAR(100) Up to 100 Characters This is a product identifier provided by you when the app is set up ∇ Developer VARCHAR(4000) Up to 4000 Characters You provided this on initial setup . ∇ Title VARCHAR(600) Up to 600 Characters You provided this when setting up the app ∇ Version VARCHAR(100) Up to 100 Characters You provided this when setting up the app ∇ Product Type Identifier VARCHAR(20) Up to 20 Characters This field defines the type of transaction (e.g. initial download, update, etc) – See Appendix E Units DECIMAL(18,2) Up to 18 Characters This is the aggregated number of units Developer Proceeds (per item) DECIMAL(18,2) Up to 18 Characters Your proceeds for each item delivered Begin Date Date Date in MM/DD/YYYY Date of beginning of report End Date Date Date in MM/DD/YYYY Date of end of report Customer Currency CHAR(3) Up to 3 Characters Three character ISO code indicates the currency the customer paid in - See Appendix H Country Code CHAR(2) Up to 2 Characters Two character ISO country code indicates what App Store the purchase occurred in – See Appendix F Currency of Proceeds CHAR(3) Up to 3 Characters Currency your proceeds are earned in – See Appendix H Apple Identifier DECIMAL(18,0) Up to 18 Characters This is Apple's unique identifier. If you have questions about a product, it is best to include this identifier. Customer Price DECIMAL(18,2) Up to 18 Characters Retail Price displayed on the App Store and charged to the customer. Promo Code VARCHAR(10) Up to 10 Characters If the transaction was part of a promotion this field will contain a value. For all non-promotional items this field will be blank - See Appendix G Parent Identifier VARCHAR(100) Up to 100 Characters For In-App Purchases this will be populated with the SKU from the originating app. Subscription VARCHAR(10) Up to 10 Characters This field defines whether an autorenewable subscription purchase is a new purchase or a renewal. See Appendix I. Period VARCHAR(30) Up to 30 Characters This field defines the duration of an auto-renewable subscription purchase. See Appendix I. ∇ Apple generally does not modify this field. What you provided when setting up your app is passed through on the report. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 20Appendix B - Opt-In Report Field Definitions The definitions apply to Weekly Opt-In Reports. Report Field Report Data Type Values Notes First Name VARCHAR(100) Up to 100 Characters First Name of Customer Last Name VARCHAR(100) Up to 100 Characters Last Name of Customer Email Address VARCHAR(100) Up to 100 Characters Email Address of Customer Postal Code VARCHAR(50) Up to 50 Characters Postal Code of Customer Apple Identifier DECIMAL(18,0) Up to 18 Characters This is Apple's unique identifier. If you have questions about a product, it is best to include this identifier. Report Start Date DATE Date in MM/DD/YYYY Date of beginning of report Report End Date DATE Date in MM/DD/YYYY Date of end of report Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 21Appendix C - Apple Fiscal Calendar Monthly Financial Reports are based on Apple’s reporting calendar shown below. Months represent either four (4) or five (5) weeks (the first month of each quarter has an extra week) and the weeks run from Sunday through Saturday. All months start on Sunday and end on Saturday. Monthly reports are also distributed on iTunes Connect and available based on the contractually agreed timeframes. Sales and Trends reports are generated using different time frames and represent near immediate feedback of transactions. Finance Reports are based on customer invoicing and financial processing. Reconciliation between the reports is not recommended due to the timing and reporting differences. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 22Appendix D - Definition of Day and Week What is a Day? 12:00:00 AM to 11:59:59 PM in the time zone used for that territory (see table below). What is a Week? Monday 12:00:00 AM to Sunday 11:59:59 PM What time is the report date based on? Territory Time Zone US, Canada, Latin America Pacific Time (PT) Europe, Middle East, Africa, Asia Pacific Central Europe Time (CET) Japan Japan Standard Time (JST) Australia, New Zealand Western Standard Time (WST) When are reports available? Reports are generated after the close of business in the final time zone (which is PT). As such, all reports are generally available by 8:00 AM PT for the prior day or week. Earlier access to reporting for other time zones (where the close of business is earlier) is not available. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 23Appendix E – Product Type Identifiers Product Type Identifier Type Description 1 Free or Paid Apps iPhone and iPod Touch, iOS 7 Updates iPhone and iPod Touch, iOS IA1 In Apps Purchase, iOS IA9 In Apps Subscription, iOS IAY In Apps Auto-Renewable Subscription, iOS IAC In Apps Free Subscription, iOS 1F Free or Paid Apps Universal, iOS 7F Updates Universal, iOS 1T Free or Paid Apps iPad, iOS 7T Updates iPad, iOS F1 Free or Paid Apps Mac OS F7 Updates Mac OS FI1 In Apps Mac OS 1E Paid Apps Custom iPhone and iPod Touch, iOS 1EP Paid Apps Custom iPad, iOS 1EU Paid Apps Custom Universal, iOS Dashboard Types Type Product Type Identifier Description Free Apps 1, 1F, 1T, F1 Where price = ‘0’ Paid Apps 1, 1F, 1T, F1 Where price > ‘0’ In Apps IA1, IA9, IAY. FI1 Updates 7, 7F, 7T, F7 Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 24Appendix F – Country Codes Country Code Country Name Country Code Country Name Country Code Country Name AE United Arab Emirates GD Grenada NG Nigeria AG Antigua and Barbuda GH Ghana NI Nicaragua AI Anguilla GR Greece NL Netherlands AM Armenia GT Guatemala NO Norway AO Angola GY Guyana NZ New Zealand AR Argentina HK Hong Kong OM Oman AT Austria HN Honduras PA Panama AU Australia HR Croatia PE Peru AZ Azerbaijan HU Hungary PH Philippines BB Barbados ID Indonesia PK Pakistan BE Belgium IE Ireland PL Poland BG Bulgaria IL Israel PT Portugal BH Bahrain IN India PY Paraguay BM Bermuda IS Iceland QA Qatar BN Brunei IT Italy RO Romania BO Bolivia JM Jamaica RU Russia BR Brazil JO Jordan SA Saudi Arabia BS Bahamas JP Japan SE Sweden BW Botswana KE Kenya SG Singapore BY Belarus KN St. Kitts and Nevis SI Slovenia BZ Belize KR Republic Of Korea SK Slovakia CA Canada KW Kuwait SN Senegal CH Switzerland KY Cayman Islands SR Suriname CL Chile KZ Kazakstan SV El Salvador CN China LB Lebanon TC Turks and Caicos CO Colombia LC St. Lucia TH Thailand CR Costa Rica LK Sri Lanka TN Tunisia CY Cyprus LT Lithuania TR Turkey CZ Czech Republic LU Luxembourg TT Trinidad and Tobago DE Germany LV Latvia TW Taiwan DK Denmark MD Republic Of Moldova TZ Tanzania DM Dominica MG Madagascar UG Uganda DO Dominican Republic MK Macedonia US United States DZ Algeria ML Mali UY Uruguay EC Ecuador MO Macau UZ Uzbekistan EE Estonia MS Montserrat VC St. Vincent and The Grenadines EG Egypt MT Malta VE Venezuela ES Spain MU Mauritius VG British Virgin Islands FI Finland MX Mexico VN Vietnam FR France MY Malaysia YE Yemen GB United Kingdom NE Niger ZA South Africa Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 25Appendix G – Promotional Codes The promo code field contains different values depending on the type of promotion. The following definitions describe the possible values that may appear in the field other than null (null means the item is a standard transaction). Only one value is possible per line in the report: Promo Code Description CR - RW Promotional codes where the proceeds have been waived (The customer price will be 0 and the proceeds will be 0). These transactions are the result of iTunes Connect Developer Code redemptions. GP Purchase of a Gift by the giver GR Redemption of a Gift by the receiver EDU Education Store transaction Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 26Appendix H – Currency Codes Currency Code Currency Country AUD Australian Dollar CAD Canadian Dollar CHF Swiss Franc DKK Danish Kroner EUR European Euro GBP British Pound JPY Japanese Yen MXN Mexican Peso NOK Norwegian Kroner NZD New Zealand Dollar SEK Swedish Kronor USD United States Dollar Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 27Appendix I - Subscription and Period Field Values The Subscription field indicates whether the auto-renewable subscription purchase is a new purchase or a renewal. Subscription Field Value New Renewal The Period field indicates the duration of the auto-renewable subscription purchase or renewal. Period Field Value 7 Days 1 Month 2 Months 3 Months 6 Months 1 Year Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 28Appendix J - FAQs What does each column represent in my reports? Please refer to Appendix A and B. I am seeing differences between Financial Reports and Sales and Trends reports, why? The daily and weekly reports are based on customer interaction (clicks) and are coming from real-time systems while the monthly reports are based on settled financial transactions and are coming from our financial systems. There are intentional differences in processing and time-frames between those two types of reports. For example, the weekly reports are from Monday to Sunday, while the Financial Reports are based on the Apple Fiscal Calendar and always end on Saturday. Reconciliation between the reports is not recommended due to the timing and reporting differences. Do weekly reports reconcile with the daily reports? Yes. Both daily and weekly reports are coming from the same system and they are based on customer interaction (clicks). They will reconcile. I see a high volume of sales for a short period of time (could be up to a week) and the sales drop down, what does this mean? It is very common that some items get a high amount of sales for a short period of time and the numbers get back to normal. It is generally due to a particular promotion related with a web blog or a sales campaign that includes an item that might be associated with iTunes or the content. There is also a very common case where a content's sales drop to zero.  In this case, this might be an indication of content being unavailable in iTunes due to number of reasons. I don’t see any sales for a particular item, why? This can be an indication of an item not being available in the store for different reasons. Check the product availability in iTunes Connect and ensure that the latest contracts are agreed to and in place. How can I identify refunds? Sales and Trends reports expose refunds to allow you to monitor refund rate by product. You will see a negative unit value for refund transactions. Why there are refunds on my reports? We will provide a refund if the customer experience was in our opinion unsatisfactory (generally quality issues). One thing you can monitor on your reports is the rate of refunds and the content that is refunded since it is an indication of quality issues with your content. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 29Appendix K - Sample Sales Report The following is a sample Sales report to help you interpret its contents. Price fields are dependent on the storefront 1 from which the customer purchases the app, and the price of the app at the time of purchase 2 . (For complete field definitions see Appendix A) Reading the Report The example above is the most likely scenario you will see: ■ SKU – “SKU1” is the SKU attached to this app by the developer. ■ Developer – “Vendor” is the name that the app is sold under on the store ■ Title – “App-1” is the name of the app ■ Product Type Identifier – “1” denotes the type of transaction (initial download) ■ Units – “352” is the number of units sold for a given day/week ■ Developer Proceeds – “3.65” is proceeds, net of commission, you will receive for each sale of the app ■ Customer Currency – “GBP” (Great Britain Pounds) is the currency in which the customer purchased the app ■ Currency of Proceeds – “GBP” (Great Britain Pounds) is the currency in which your proceeds were earned for the app ■ Customer Price – “5.99” is the price paid by the customer for the app Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 30 1 As new territories are added, storefronts will further differentiate records 2 If you change your price during the reporting period, the report will show multiple price points for the same countryAdditional Reporting Scenarios We have provided some additional scenarios and sample extract to help you further understand your reports. In your reports the Product Type Identifier denotes the type of transaction (See Appendix E for a list of all types). The Product Type Identifier must be taken into account in all of the following scenarios. Scenario 1 (Product Type Identifier=1; Units=16; Developer Proceeds=4.86) This is similar to the first line; the Developer Proceeds value will always be greater than zero for all paid apps and zero for free apps. Scenario 2 (Product Type Identifier=7; Units=1; Developer Proceeds=0) Certain line items will have 0 in the Developer Proceeds field. Even if you only have paid apps on the store, the Developer Proceeds will be 0 for all updates (Product Type Identifier = 7). Scenario 3 (Product Type Identifier=1; Units=-1; Developer Proceeds=7; Customer Price=-9.99) You may see negative units when a customer returns a product. All returns will have a Product Type Identifier of 1 and both Units and Customer Price will be a negative value. Refer to Appendix J for additional information on returns. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 31Appendix L – Other Uses Below you will see some sample ideas that the data can be used for on a daily basis. 1. Business Health Monitoring By tracking volume of sales per unit or revenue, the health of your business can be tracked. Sudden drop in sales may indicate issues such as top seller being not available any more. 2. Content Quality Issues By tracking the refunds, you can identify and replace the asset that is being refunded to the customer if the refunds are specific to one or more items. Typical ratio of refunds to overall sales shall not exceed 0.10%. 3. Pricing Issues When organizations get larger, it is always challenging to have fast/efficient communication between the operational teams that are providing the metadata to iTunes and the Management, Marketing, Finance and Business Development team. Tracking pricing will indicate any disconnect between different groups and will provide opportunity to fix issues sooner and minimize the impact. 4. Price Elasticity We believe that careful management of price can increase your sales. By using the reports you can monitor percent change in sales in correlation with a percent change in customer price. If applied correctly this type of analysis will help you set the best price for your product to maximize your revenue. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 32Appendix M - Newsstand Report Field Definitions The definitions apply to Newsstand reports. Report Field Report Data Type Values Notes Provider CHAR(5) - APPLE Up to 5 Characters The service provider in your reports will typically be Apple Provider Country CHAR(2) - US Up to 2 Characters The service provider country code will typically be US SKU VARCHAR(100) Up to 100 Characters This is a product identifier provided by you when the app is set up Developer VARCHAR(4000) Up to 4000 Characters You provided this on initial setup. Title VARCHAR(600) Up to 600 Characters You provided this when setting up the app Version VARCHAR(100) Up to 100 Characters You provided this when setting up the app Product Type Identifier VARCHAR(20) Up to 20 Characters This field defines the type of transaction (e.g. initial download, update, etc) – See Appendix E Units DECIMAL(18,2) Up to 18 Characters This is the aggregated number of units Developer Proceeds (per item) DECIMAL(18,2) Up to 18 Characters Your proceeds for each item delivered Customer Currency CHAR(3) Up to 3 Characters Three character ISO code indicates the currency the customer paid in - See Appendix H Country Code CHAR(2) Up to 2 Characters Two character ISO country code indicates what App Store the purchase occurred in – See Appendix F Currency of Proceeds CHAR(3) Up to 3 Characters Currency your proceeds are earned in – See Appendix H Apple Identifier DECIMAL(18,0) Up to 18 Characters This is Apple's unique identifier. If you have questions about a product, it is best to include this identifier. Customer Price DECIMAL(18,2) Up to 18 Characters Retail Price displayed on the App Store and charged to the customer. Promo Code VARCHAR(10) Up to 10 Characters If the transaction was part of a promotion this field will contain a value. For all non-promotional items this field will be blank - See Appendix G Parent Identifier VARCHAR(100) Up to 100 Characters For In-App Purchases this will be populated with the SKU from the originating app. Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 33Subscription VARCHAR(10) Up to 10 Characters This field defines whether an auto-renewable subscription purchase is a new purchase or a renewal. See Appendix I. Period VARCHAR(30) Up to 30 Characters This field defines the duration of an auto-renewable subscription purchase. See Appendix I. Download Date (PST) TIMESTAMP(0) Date in MM/DD/YYYY Download Date Customer Identifier DECIMAL(18,0) Up to 18 Characters Customer Identification Report Date (Local) DATE Date in MM/DD/YYYY Report Date Sales/Return CHAR(1) Up to 1 character S or R; R is always a refund, R is not a reversal Apple Inc. iTunes Connect Sales and Trends Guide, App Store Version 5.3  iTunes Connect Sales and Trends Guide, App Store (Version 5.3, August 2012) 34 Secure Coding GuideContents Introduction to Secure Coding Guide 7 At a Glance 7 Hackers, Crackers, and Attackers 7 No Platform Is Immune 8 How to Use This Document 9 See Also 10 Types of Security Vulnerabilities 11 Buffer Overflows 11 Unvalidated Input 12 Race Conditions 13 Interprocess Communication 13 Insecure File Operations 13 Access Control Problems 14 Secure Storage and Encryption 15 Social Engineering 16 Avoiding Buffer Overflows and Underflows 17 Stack Overflows 18 Heap Overflows 20 String Handling 22 Calculating Buffer Sizes 25 Avoiding Integer Overflows and Underflows 27 Detecting Buffer Overflows 28 Avoiding Buffer Underflows 29 Validating Input and Interprocess Communication 33 Risks of Unvalidated Input 33 Causing a Buffer Overflow 33 Format String Attacks 34 URLs and File Handling 36 Code Insertion 37 Social Engineering 37 Modifications to Archived Data 38 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 2Fuzzing 39 Interprocess Communication and Networking 40 Race Conditions and Secure File Operations 43 Avoiding Race Conditions 43 Time of Check Versus Time of Use 44 Signal Handling 46 Securing Signal Handlers 46 Securing File Operations 47 Check Result Codes 47 Watch Out for Hard Links 48 Watch Out for Symbolic Links 49 Case-Insensitive File Systems Can Thwart Your Security Model 49 Create Temporary Files Correctly 50 Files in Publicly Writable Directories Are Dangerous 51 Other Tips 57 Elevating Privileges Safely 59 Circumstances Requiring Elevated Privileges 59 The Hostile Environment and the Principle of Least Privilege 60 Launching a New Process 61 Executing Command-Line Arguments 61 Inheriting File Descriptors 61 Abusing Environment Variables 62 Modifying Process Limits 62 File Operation Interference 63 Avoiding Elevated Privileges 63 Running with Elevated Privileges 63 Calls to Change Privilege Level 64 Avoiding Forking Off a Privileged Process 65 authopen 65 launchd 66 Limitations and Risks of Other Mechanisms 67 Factoring Applications 69 Example: Preauthorizing 69 Helper Tool Cautions 71 Authorization and Trust Policies 72 Security in a KEXT 72 Designing Secure User Interfaces 73 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 3 ContentsUse Secure Defaults 73 Meet Users’ Expectations for Security 74 Secure All Interfaces 75 Place Files in Secure Locations 75 Make Security Choices Clear 76 Fight Social Engineering Attacks 78 Use Security APIs When Possible 79 Designing Secure Helpers and Daemons 81 Avoid Puppeteering 81 Use Whitelists 82 Use Abstract Identifiers and Structures 82 Use the Smell Test 83 Treat Both App and Helper as Hostile 83 Run Daemons as Unique Users 84 Start Other Processes Safely 84 Security Development Checklists 86 Use of Privilege 86 Data, Configuration, and Temporary Files 88 Network Port Use 89 Audit Logs 91 Client-Server Authentication 93 Integer and Buffer Overflows 97 Cryptographic Function Use 97 Installation and Loading 98 Use of External Tools and Libraries 100 Kernel Security 101 Third-Party Software Security Guidelines 103 Respect Users’ Privacy 103 Provide Upgrade Information 103 Store Information in Appropriate Places 103 Avoid Requiring Elevated Privileges 104 Implement Secure Development Practices 104 Test for Security 104 Helpful Resources 105 Document Revision History 106 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 4 ContentsGlossary 107 Index 110 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 5 ContentsFigures, Tables, and Listings Avoiding Buffer Overflows and Underflows 17 Figure 2-1 Schematic view of the stack 19 Figure 2-2 Stack after malicious buffer overflow 20 Figure 2-3 Heap overflow 21 Figure 2-4 C string handling functions and buffer overflows 22 Figure 2-5 Buffer overflow crash log 29 Table 2-1 String functions to use and avoid 23 Table 2-2 Avoid hard-coded buffer sizes 25 Table 2-3 Avoid unsafe concatenation 26 Race Conditions and Secure File Operations 43 Table 4-1 C file functions to avoid and to use 55 Elevating Privileges Safely 59 Listing 5-1 Non-privileged process 70 Listing 5-2 Privileged process 71 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 6Secure coding is the practice of writing programs that are resistant to attack by malicious or mischievous people or programs. Secure coding helps protect a user’s data from theft or corruption. In addition, an insecure program can provide accessfor an attacker to take control of a server or a user’s computer, resulting in anything from a denial of service to a single user to the compromise of secrets, loss of service, or damage to the systems of thousands of users. Secure coding is important for all software; if you write any code that runs on Macintosh computers or on iOS devices, from scripts for your own use to commercial software applications, you should be familiar with the information in this document. At a Glance Every program is a potential target. Attackers will try to find security vulnerabilities in your applications or servers. They will then try to use these vulnerabilities to steal secrets, corrupt programs and data, and gain control of computer systems and networks. Your customers’ property and your reputation are at stake. Security is notsomething that can be added to software as an afterthought; just as a shed made out of cardboard cannot be made secure by adding a padlock to the door, an insecure tool or application may require extensive redesign to secure it. You must identify the nature of the threats to your software and incorporate secure coding practices throughout the planning and development of your product. This chapter explains the types of threatsthat yoursoftware may face. Other chaptersin this document describe specific types of vulnerabilities and give guidance on how to avoid them. Hackers, Crackers, and Attackers Contrary to the usage by most news media, within the computer industry the term hacker refers to an expert programmer—one who enjoyslearning about the intricacies of code or an operating system. In general, hackers are not malicious. When most hackers find security vulnerabilities in code, they inform the company or organization that’s responsible for the code so that they can fix the problem. Some hackers—especially if they feel their warnings are being ignored—publish the vulnerabilities or even devise and publish exploits (code that takes advantage of the vulnerability). 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 7 Introduction to Secure Coding GuideThe malicious individuals who break into programs and systems in order to do damage or to steal something are referred to as crackers, attackers, or black hats. Most attackers are not highly skilled, but take advantage of published exploit code and known techniques to do their damage. People (usually, though not always, young men) who use published code (scripts) to attack software and computer systems are sometimes called script kiddies. Attackers may be motivated by a desire to steal money, identities, and othersecretsfor personal gain; corporate secrets for their employer’s or their own use; or state secrets for use by hostile governments or terrorist organizations. Some crackers break into applications or operating systems just to show that they can do it; nevertheless, they can cause considerable damage. Because attacks can be automated and replicated, any weakness, no matter how slight, can be exploited. The large number of insiders who are attacking systems is of importance to security design because, whereas malicious hackers and script kiddies are most likely to rely on remote access to computers to do their dirty work, insiders might have physical access to the computer being attacked. Your software must be resistant to both attacks over a network and attacks by people sitting at the computer keyboard—you cannot rely on firewalls and server passwords to protect you. No Platform Is Immune So far, OS X has not fallen prey to any major, automated attack like the MyDoom virus. There are several reasons for this. One is that OS X is based on open source software such as BSD; many hackers have searched this software over the years looking for security vulnerabilities, so that not many vulnerabilities remain. Another is that the OS X turns off all routable networking services by default. Also, the email and internet clients used most commonly on OS X do not have privileged access to the operating system and are less vulnerable to attack than those used on some other common operating systems. Finally, Apple actively reviewsthe operating system and applications for security vulnerabilities, and issues downloadable security updates frequently. iOS is based on OS X and shares many of its security characteristics. In addition, it is inherently more secure than even OS X because each application is restricted in the files and system resources it can access. Beginning in version 10.7, Mac apps can opt into similar protection. That’s the good news. The bad news is that applications and operating systems are constantly under attack. Every day, black hat hackers discover new vulnerabilities and publish exploit code. Criminals and script kiddies then use that exploit code to attack vulnerable systems. Also, security researchers have found many vulnerabilities on a variety of systems that, if exploited, could have resulted in loss of data, allowing an attacker to steal secrets, or enabling an attacker to run code on someone else’s computer. Introduction to Secure Coding Guide At a Glance 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 8A large-scale, widespread attack is not needed to cause monetary and other damages; a single break-in is sufficient if the system broken into contains valuable information. Although major attacks of viruses or worms get a lot of attention from the media, the destruction or compromising of data on a single computer is what matters to the average user. For your users’sake, you should take every security vulnerability seriously and work to correct known problems quickly. If every Macintosh and iOS developer followsthe advice in this document and other books on electronic security, and if the owner of each Macintosh takes common-sense precautions such as using strong passwords and encrypting sensitive data, then OS X and iOS will maintain their reputationsfor being safe, reliable operating systems, and your company’s products will benefit from being associated with OS X or iOS. How to Use This Document This document assumes that you have already read Security Overview. The document begins with “Types of Security Vulnerabilities” (page 11), which gives a brief introduction to the nature of each of the types of security vulnerability commonly found in software. This chapter provides background information that you should understand before reading the other chapters in the document. If you’re not sure what a race condition is, for example, or why it poses a security risk, this chapter is the place to start. The remaining chapters in the document discuss specific types of security vulnerabilities in some detail. These chapters can be read in any order, or as suggested by the software development checklist in “Security Development Checklists” (page 86). ● “Avoiding Buffer Overflows And Underflows” (page 17) describes the various types of buffer overflows and explains how to avoid them. ● “Validating Input And Interprocess Communication” (page 33) discusses why and how you must validate every type of input your program receives from untrusted sources. ● “Race Conditions and Secure File Operations” (page 43) explains how race conditions occur, discusses ways to avoid them, and describes insecure and secure file operations. ● “Elevating Privileges Safely” (page 59) describes how to avoid running code with elevated privileges and what to do if you can’t avoid it entirely. ● “Designing Secure User Interfaces” (page 73) discusses how the user interface of a program can enhance or compromise security and gives some guidance on how to write a security-enhancing UI. ● “Designing Secure Helpers And Daemons” (page 81) describes how to design helper applications in ways that are conducive to privilege separation. Introduction to Secure Coding Guide How to Use This Document 2012-06-11 | © 2012 Apple Inc. All Rights Reserved. 9In addition, the appendix “Security Development Checklists” (page 86) provides a convenient list of tasks that you should perform before shipping an application, and the appendix “Third-Party Software Security Guidelines” (page 103) provides a list of guidelines for third-party applications bundled with OS X. See Also This document concentrates on security vulnerabilities and programming practices of special interest to developers using OS X or iOS. For discussions of secure programming of interest to all programmers, see the following books and documents: ● See Viega and McGraw, Building Secure Software , Addison Wesley, 2002; for a general discussion of secure programming, especially as it relates to C programming and writing scripts. ● SeeWheeler, Secure Programming for Linux andUnixHOWTO, available athttp://www.dwheeler.com/secureprograms/; for discussions ofseveral types ofsecurity vulnerabilities and programming tipsfor UNIX-based operating systems, most of which apply to OS X. ● See Cranor and Garfinkel, Security and Usability: Designing Secure Systems that People Can Use , O’Reilly, 2005