>Top 0. Preface:
- Change of society and business environment is called 'context' herein.
- There are serous troubles in software development based on the waterfall type (WF) in Japan.
- Recent rapid changing business environment required non-waterfall type (NWF) software development, which would be suitable for such changing environment, or business context.
- Though NWF software development is not so much penetrated in Japan, but not a few success stories are getting reported.
- Agile SW development has related to the lean production methodology in 1980s applied in Japanese automobile industry.
- Lean SW development is also proposed in US in 2003, aiming; 1) minimize uselessness, 2) keeping qualification of SW, and 3) total optimization, as well as paying attention to the developing peoples thinking quick delivery of SW products.
- Now LEfSE (Lean Enablers for Systems Engineering) is considered at INCOSE (Lean Enablers for Systems Engineering) in US.
- 一方で、NWF (Non-Waterfall) ソフトウェア開発による成功事例も増えてきている。
>Top 1. Background of NWF software development:
- NWF SW development has following features:
- appropriate response to changing business needs:
rapid SW development is required to meet with rapid change of business
- solution against WF SW development:
all RFP is decided before SW development, which may cause discrepancy or delay of SW completion, or the completed SW become obsolete at the cut over. Also, frequent returning of the development process consume useless time and money.
- solution for the structural problems in SW industry:
The present SW industry contains multi-layered
sub-contractors structure, which causes degraded morale or less achievement goals of SW developers.
- Favorable area of NFW SW development:
- the changing business paradigm: shortening of TTM (Time to Market)
- global competition
- frequently changing SW functions (Web services, package development, etc.)
- visible achievement of each SW engineer involved
- Definition of NWF SW development:
Agile Manifesto (Manifesto for Agile Software Development) by 17 developers in 2001, which values:
- SW development other than WF SW development defined by Winston W. Royce in 1970 "Managing the Development of Large Software Systems."
- As the examples of NWF development are:
- Prototype: Frederick P. Brooks, 1975
- Spiral: Barry w. Boehm, 1988
- RAD (Rapid Application Development): James Martin, 1991
- RUP (Rational Unified Process): Philippe Kruchten, 2000
- EVO (Evolutionary Development): T. Gilb, 1981
- Scrum (Agile Software Scrum): Ken Schwaber, 1993
- DSDM: DSDM Ver-1, 1995
- XP: eXtreme Programming: Knet Beck, 1996
- FDD (Feature Driven Development): Peter Coad, 1997
- LSD (Lean Software Development): Mary & Tom Poppendieck, 2002
- Crystal Clear: Alistair Cockburn, 2004
- EssUp (Essential UP): Ivar H. Jabobson, 2005
- ASD (Adaptive Software Development)
- MDA (Model Driven Architecture)
- Kanban: David Anderson, 2010
Trial and error areas of NWF SW development:
- More human dialogue than process or tool
(Face-to-face communication is most effective to enhance team capability) usually no more than 10 people team.
- More functioning SW than overall documentation
(cycles within a few weeks or a few months.)
usually 2-4 week iterations
- More collaboration with client than contract negotiation
(works collocation with the client and developer)
- More responsive to the changes than keeping the original plan.
(best architecture is made from autonomous team; simpleness matters.)
Viewpoint from User and Developer of NWF SW:
- Large scale SW development requiring more than 10 developers
- Distributed or offshore development: multi location and time difference are bottleneck.
- Development beyond organization or company:
- Embedded software: version up after release is difficult.
Viewpoint from Developer:
- quick confirmation by moving software
- quick start of SW development before finalizing RFP
- flexible to the change of priority or RFP
- avoidance of useless or obsolete function during development.
Premises of both U and V sides:
- quick confirmation of the user's merit
- short cycle of development can accumulate development experiences
- maintenance of motivation by achievement.
Relation between Agile SW development and Employment:
- Vender understands the User's business.
- User participated development team and makes timely decision making
- Both V/U allows changes of scope.
- Agile SW development, in-house SW development, and liquidity of employment relates each other, particularly in Japan.
- Cloud age changes the ownership of SW development resources; human resources for in-house development is getting cost or risk factor.
- Former in-house SW development like major steel companies experienced similar SW development environment like Agile development.
- Offshore SW development supplies enormous human developers, but the highly skilled engineers are still short in supply, which is required for Agile development.
1. NWF SW開発の背景:
- Lean SW development
- Crystal Clear
- 動くSW：2-3週間〜2-3ヶ月; 通常2-4週間のイテレーション
- 分散, Offshore拠点
>Top 2. Paradigm shift of Owner:
- Paradigm shift for corporate executives:
- Active role of CEO/COO/CIO is required, particularly for Agile development.
- IT governance and Corporate governance are closely related (like as internal control and corporate control)
- IT issues are getting to be the issue of executives, which could be a crime of omission.
- NWF SW development relates closely to:
- end user computing
- IT risk evaluation
- maintenance and operation of IT systems
- IT systems troubles
- project management
- outsourced assignment control
- What is the system?
- cluster of SW
- IT system is SW life cycle process, showing performance more than the sum of components; (Holonisum)
- system is autonomous and has fractal features (because SW itself is a system).
- In case of system troubles, causing the problem which the system itself created.
- Thus to solve such system trouble, it is needed to change the whole system.
- The system comprise HW and SW; HW can be estimated exactly its behavior, but SW is not, because which is man-made product.
- In making a IT system, there are 'Owner', 'Designer', and 'Constructor'
- IT System Cycle:
- IT system should be like a town development which needs administrator (CIO) to maintain its function, continuing crating corporate values (or corporate culture)
- This long-term framework should be Enterprise Architecture (EA).
- Corporate business is on-going; just like enlarging a ship while sailing.
- Royce never uses a term of 'Waterfall,' on the contrary he admitted one or twice return of the process (Build Twice!)
- If there were no return of process, it put the problems later stages or making simple mass production.
- In Boem's spiral models, confirmation by prototyping is needed to proceed the next stage.
- SW development in 1970's was mostly report design; by since 1980's SW is used on-site, which means 'market' factor become a component of SW.
- NW computing compose of HW, OS, DBMS, and NW which are collectively called 'Cloud' become now prevailed.
- Former SW should keep QCD; Quality (less bug), Cost (cheap), and D (on time).
- But recent SW is required EEE (Efficacy; correct answer), Efficiency, and Effectiveness; in a short which should be effective for the business object (like response time, or inventory turnover)
- Large scale SW development by NWF model:
- Burden of owner increases, participating qualified staff to the project teams.
- Hierarchical formation such as Scrum of Scrums or meta-Scrum team is needed to adjust the large scale SW development.
- Analogy with construction industry:
- In 1980's, general contractors in construction industry focussed technology-oriented building of skyscrapers and gigantic buildings, paying limited attention to the environment and city view. (People tend to behave according to the farming database of legal & technical restriction, preference, and cost)
- Since 2000, new trend of designing process (Ultra Liner Process) is proposed to respond to such social requirement; the process should be 1) not jump, 2) not divide into branches, and 3) not return the former steps. Also making models to keep the logs are needed to proceed steps. This process looks like waterfall liner model, but also it resembles to agile process.
- Intimate communication with owner using such models is also effective in NWF SW development.
- Value of NWF SW development for owner:
- NWF SW development has unique problems or risk except for WF development.
- It is important to start from the project which would be most appropriate by developing NWF type.
- Client (End-user, EU): who bears all costs related to IT services and promote business effectively using IT system.
- IT Division; who supplies IT service for their clients. Usually IT Division and Client belong to the same company or group, and generally use outsource. SMB may have only a few staff and outsourced personnel.
- There are two types of SW development:
- Reliance-centric SW: ERP, SaaS
- Agility-centric SW: SW used for new Web & mobile-using business. RFP could not be determined preemptively due to business reason. (Business requirement is flexible and chargeable depending on the situation.)
- 特にNWF SW開発の場合
- 情報システムとは何か :
- System Lifecycle Processに関わる部分
- 自己組織化・フラクタル性を有する (SWも一つのシステム故)
- Owner (CEO/COO)はSW利用により企業価値と企業文化を生み続ける必要あり。
- そのために小さな保守を繰り返して維持する (都市計画の如く)
- 大規模開発にはScrum of Scrumsの様な階層的組織が必要。
- SW開発には２種類ある。信頼性重視のシステム (ERPなど)と、俊敏性重視のシステム (Web,モバイル活用など)。後者は、ビジネス上の理由で要件を予め決められない。
>Top 3. Overseas Agile SW development
- Forrester Research, Inc published in 2010, that Agile method has become major in SW development, mentioning, Agile development 35% vs. Waterfall 13% as of 3Q inn 2009; also hybrid development at site to meet with large project is challenging.
- Share of Agile development method: Scrum 52%, Scrum+XP Hybrid 14%, Custom Hybrid 9%.
- # of qualified Scrum (CSM+CSPO+CSP) engineers; US 67,000; UK
- Case of Denmark：
DBT (Danish Board of Technology) issues a warning report to avoid failure of IT project, and strongly recommends to use NWF SW development. Essence of the report is as follows;
- To start the project at minimum scale, say, developers be no more than 30 and the first delivery be no more than 6 months.
- To contract be flexible and changeable.
- To include incentive clause than penalty clause.
- To include communication plan covering all scope of project.
- To make participate of the end user at full-time basis and define their function.
- To clarify the function of user and vendor respectively, and review their collaboration (retrospective) at each end of the iteration (sprint).
- and so on.
- It is no exaggeration to say the the entire global IT industry is experiencing an agile wave. The philosophy is summarized in the following table (from the Manifesto for Agile SW Development)
|processes and tools
||individuals and interaction
||collaboration with the customer
|following a plan
||adapting to changes
- The agile method are a reaction to the processes that look good in theory but that do not hold up in practice. The agile methods are therefore described as empirical - they are base entirely on practical experiences and work methods that are proven to work.
- Where older methods are predictive and attempt to foresee future meeds, the agile methods are adaptive and quickly adapt to new demands, adhering to the "Embrace change!" motto. The only measurement of success is functioning products.
- Another important principle is simplicity and lean thinking.
- <Scrum> deals with how the project is organized and planned. Experience shows that 30 days (about 1000 effective hours for a group) give a comfortable work pace and adaptability.
- For larger project, so-called Scrum of Scrums can include hundred of programmers, organized in dozens of Scrum Teams.
- Scrum is a rugby team for the close-knit shoulder-to-shoulder formation a rugby team forms to jointly move the ball forward. The word is first used by Takeuch and Nonaka in a famous article published in HBR describing the most successful product development projects in Japan.
- <XP> deals with how to work with programing; such as pair programming and test case production before coding.
- <Lean Development> stems from the manufacturing industry's Just-In-Time and Lean Production concepts. It deals more with how to organize the entire company's development activities at management level.
- <Scrum>は、特に組織運営を重視。30日毎のスプリント (1000人時の作業量)が有効。
- 大型案件の場合は、Scrum of Scrumsを編成。
- <XP> はプログラミングの方法論。ペア・プログラミングやコーディング前のテストケースなどが特徴。
<Scrum formation> Re: The new new product development game, , by H. Takeuchi & I. Nonaka, HBR
- Under the rugby approach, the product development process emerges from the constant interaction of a hand-picked, multidisciplinary team whose members work together from start to finish. Rather than moving in defined, highly structured stages, the process in born out of the team members interplay.
- A group of engineers may start to design the product (Phase-3) before all the results of the feasibility tests (Phase-2) are in. Or, the team may be forced to reconsider a decision as a result of later information.. The team does not stop then, but engages in iterative experimentation. This goes on in even the latest phases of the development process.
- The exhibit illustrates the difference between the traditional, linear approach to product development and the rugby approach.
- This approach is essential for company seeking to develop new products quickly and flexibly. The shift from a linear to an integrated approach encourages trial and error and challenges the status quo. It stimulates new kinds of learning and thinking within the organization at different levels and functions.
>Top 4. Glossary of Scrum, etc.:
- Adaptive: adjustable that project goals or schedule are adjusted in line with how the external factors change.
- Burn-down Chart: a diagram that monitors how much work remains to implement a segment of the SW being developed during a Sprint.
- Daily Scrum; brief, daily meeting ( about 15 min.) between the Scrum Master and the Scrum Team. The purpose is to keep work flowing smoothly and eliminate any impediments.
- Empirical: based on experience.
- Agile development: a methodology for SW development which emphasized adaptability, short paths between ideas and implementation, and simplified forms of collaboration.
- Sprint Retrospective: meeting (about 3 hours) held after each Sprint. The Scrum Master and the Scrum Team review both what went well and what should be improved in the next Sprint.
- Predictive: foresighted, in this context, project goals and schedules based on a prognosis of external factors made at the beginning of the project.
- Product Backlog: current "to-do list" that contains the project's goals and priorities.
- Product Owner: the person responsible for the product's Product Backlog and that the project is working with the right things from business perspective.
- Release Backlog: the same as a Product Backlog, but restricted to a release of the product.
- Scrum Master: the team leader for the Scrum Team. But the leader doesn't decide alone.
- Scrum Team: the work force, in this case SW designers in a Scrum project. Organized its work itself and lacks a formal group manager.
- Sprint: the iteration comprised of normally 30 days during which the Scrum Team concentrates on realizing the goals defined by the project's current Sprint Backlog.
- Sprint Backlog: a to-do list for a Sprint. Consist of the assignments that the Product Owner has defined as having the highest priority. Is given its final structure during the Sprint's first day at a meeting between the Product Owner and the Scrum Team.
- Sprint Review: an informal meeting (about 4 hours) at the end of a Sprint during which the team presents and demonstrates, if relevant, for management, customers and the Product Owner what has been created during the Sprint.
- Timebox: a period during which something is to be carried out. A Sprint is a result of timebox thinking. Deadlines may not be exceeded - parts of the assignment are deleted instead.
- Story: XP: memorandum for user written on B6 card, containing simple expression. The details, if needed, are prepared separately. The development scope is decided by the team using this story card.
- YAGNI (You Ain't Gonna Need It): XP: It is important to maintain simple design and simple architecture.
4. Scrumなどの用語集 :
- Burn-down Chart：作業残務チャート
- Daily Scrum：毎日のスクラム会議
- Agile development：アジャイル開発
- Sprint Retrospective：スプリントの反省事項
- Product Backlog：作業残務量
- Product Owner：オーナー
- Release Backlog：完成済量
- Scrum Master: スクラム長
- Scrum Team: スクラムグループ開発メンバー。但しグループ長は独断しない。
- Sprint: 通常30日間のイテレーション期間
- Sprint Backlog: SprintのTo-Doリスト。オーナーとのSprint初日に提示される。
- Sprint Review: 各Sprint終了事の非公式会議(約４時間)で、このSprintの成果物を関係者に説明。
>Top 5. Preferable Contracts for NWF SW Development:
- There are two types of contract; 1) lump sum contract, and 2) consignment contract:
- <Lump sum contract - Fixed price and fixed scope>
An user asks a certain vendor to complete some software (=deliverable) within a certain budget and period. The user is to pay the lump sum price for the deliverable.
||as the requirement of the project in uncertain, the project may not be finished within the contract price.
||as the deliverable are not clearly defined, acceptability of it may be uncertain.
|unclear quality of deliverable
||The project period may be elongated, and the completion be delayed.
||the contract period may be elongated endlessly, and cannot be completed within the budget.
||overestimate may cause excessive contract price.
||reasonable profit cannot be obtained due to underestimate of the project cost.
|unclear cut over
||# members from the user side could not be decided due to unclear project period.
||cannot assign an appropriate # of engineers due to unclear project period
|collaboration of user and vendor.
||demarcation point is unclear, and in case of trouble it is unclear who is responsible.
||demarcation point between user and vendor is unclear, so the risk arisen, it is unclear who is responsible
|communication of user and vender
||instruction from user to the member from vender may be regarded as illegal.
- <Consignment contract - Time and materials>
A user asks c certain vendor to consign to complete some software within a certain budge and period. The vender agrees to try to best efforts (due diligence) to make the software
within the condition, but not necessarily commits to make it.
- Other than the above contracts, the following clause can be contracted:
- Sprint Contract: Basic contract plus additional contracts after the acceptance of each Sprint stage.
- Time and Materials with fixed scope and cost ceiling contract:
- fixed scope is not suitable for NWF development.
- Time and Materials with variable scope and cost ceiling contract:
- collaboration of U and V is important.
- Phased development contract:
- Bonus/Penalty clause:
- Fixed profit (paid after the target delivery date)
- Money for nothing, charge for free clause:
- This is Time and Material with upper limit contract price.
5. NWF SW開発にとって適切な契約:
- NWF SW開発契約：
- 派遣契約は、要員派遣を目的としたものであり、当該要員への指揮命令は (Uの企業内で作業しても)派遣先のVが持っている。
- 委託契約は、いずれの側(U/V)もいつでも契約解除可能。但し、解除条件 (催促解除も含め)を契約で明示する意味はある。
- 発注側(U)と受注側(V)の責任分岐点 (Demarcationn point)が不明確
- Project riskは全てU側
>Top 6. Why Agile development is agile?:
- Because is it iteration of small cycles method of the development?
- Probably no. Because only iteration or division of parts is equal to the total. Loss may be caused by only split.
- Because is it using efficient tool or language?
- Probably no. Even using Ruby, the efficiency is not guaranteed.
- Because is the development team compose of able and efficient engineers?
- Partly yes. Efficiency varies very much among engineers, but it requires to keep good environment not to disturb such efficiency.
- Because is it not intending to make all the scope from the beginning?
- Affirmatively yes. Agile development starts from the important and urgent portion, then gradually expand to other scopes. If the total scope is diminished, it it natural that the completion become shorter or agile.
Also, it is important to start without requiring strict completeness from the beginning. Relaxed and communicative environment also give positive impact for efficiency.
- Because is it the small team and frequently check mutually, or is it not allowed to be lazy in the team?
- Probably yes. But too much pressure each other may cause negative impact. The character and role of the team leader is also important.
- Because is it refactoring process accepted in the development team?
- Surprisingly yes. Refractoring is a process of improving or cleaning code without increasing visible function. Thus works will not be evaluated in WF development. This refactoring process is a kind of hidden investment of the future extension and maintenance of the system.
>Top 7. Herrmann Model and NWF Development:
- According to Herrmann model, our brain has following features:
- Thinking styles can be best described as a coalition of four different thinking selves (A/D are left/right neocortex, and B/C are left/right primitive limbic cortex)
- We are a product of both nature and nurture. It is the nurture aspect that predominates in who we are and who we can become.
- We have already done this with our right hand or left hand, and developed a lifetime of handedness preference and skill.
- We have developed our mental dominance into such strong preferences. We have developed language and math skills, or are exceedingly organized, or can accurately sense the feeling of others, or are always coming up with new ideas.
- Such competitive approaches can lead to better decisions and better solutions.
- NWF SW development team requires more frequent collaboration and communication between User and Vendor, which means the sense of right limbic cortex is important.
- Furthermore, the more use of right brain (both neocortex and limbic cortex) may cause creative works.
- A: 左脳の新皮質
- B: 左脳の辺縁系
- C: 右脳の辺縁系
- D: 右脳の辺縁系
- NWF SW開発には、特に右脳辺縁系の得意分野であるチームの人間関係が重要となる
>Top 8. IT Human resources:
- Source: IPA SEC, report, 2011/3
- IT Human resources belong to User company or Vendor company?
- Where IT engineers are working? (%)
- Scrum engineers (CSM=Certified Scrum Master), CSPO=Certified Scrum Product Owner, CSP (Certified Scrum Professional)
>Top 9. IPA's recommendation for NWF development:
- Realtime communication within the development team
- Publish more case studies of NWF development
- Publish contract samples based on NWF development
- Promote in-house development without contract
- Enlighten that the value of SW is a key success factor.
- Adopt J/V formation with User company and Vendor.
- Public procurement should be NFW development.
- Establish HR curriculum of wide knowledge applying to various educational institutions
- Enhance practical education (PBL= Problem Based Learning)
- Promote education through Industry- Academia collaboration
- Enhance the status of IT engineers.
- Invite more foreign students
- Increase mobility of IT personnel
- Enlarge community
- Establish training center for NWF SW development
- Publish guidelines and/or Tips of NWF SW development
- Involve qualified engineers in a NWF development team
- ユーザ企業内開発 (契約不要)
>Top 10. Why NWF SW Type couldn't proliferate in Japan?:
- I think the major reason would be that both User and Vender put emphases on keeping stable relationship on long-term basis. Most of the Japanese have considered it is valuable that the business relation should be long-term and stable one. This sentiment encompass not only business sphere, but also private life including getting job, getting married, getting to be trained, getting to keep human relations, or even selecting banks and the political party or even religious appetite; all these should be stable, unchanged, and conservative relationships.
- NWF contract aims to be most suitable in a changing environment; based on scope, delivery, human ability are changeable ones. Thus, the contract should be made on monthly iteration basis, evaluation the achievement. This may be felt by most of Japanese, a kind of dolphin show; giving bonus evaluating each time their achievement!
- Though long-term reliance and relationship is important, its evaluation and payment is different. Bonus and penalty should be fair, accountable and changeable at any time. But this visual and direct idea probably could not be acceptable both of Japanese User and Vendor.
- I'd say that both User and Vender look to be not confident, or even are afraid of having clear-cut relationship between job and payment: User could not show their clear business requirements, and Vender could not accomplish profession jobs with exact QCD. They want to be embraced each other, without asking what is your requirement and what is your professional jobs.